validation set Archives - Cracking the ML Interview!

Why don’t we tune hyper-parameters using test set and need a separate set like validation set?

Posted on February 21, 2019May 11, 2019 by InterviewBuddy

We use test set such that we build a model that generalizes well on unseen dataset. If we use test set in tuning for hyper-parameters to select the model, we’re indirectly using test set in training or rather, our model has seen the test set. Hence, it is no longer an unseen dataset but already…

Why do you need training set, test set and validation set ?

Posted on February 21, 2019May 11, 2019 by InterviewBuddy

Before any model is built for the problem in hand, the dataset exists as a single entity. One can start learning from entire dataset and use the built models to make predictions on new unseen data. The later part is called generalization in Machine Learning terminology. However, training on entire dataset available would lead to…