Training, Validation, and Test Datasets

Training, Validation, and Test Datasets#

So far, we have learned that a good machine learning model should generalize well to new, unseen data. But how do we know whether our model can actually do that? If we train and evaluate a model on the same data, the result can be misleading; the model has already seen those examples during training. To evaluate it fairly, we need to test it on data it has not seen before.

Let’s start by dividing our dataset into two parts: a training set and a test set.

Figure: The original dataset is shuffled and divided into two parts. The training set is used to train the model, while the test set is kept separate to evaluate the model on unseen data. Source: O'Reilly.

1. Training Set#

The training set is the data used to train the model. The model learns patterns and relationships from these examples. For example, if we are predicting house prices, the training data might include:

House size
Number of bedrooms
Location
Age of the house
Actual selling price

The model uses these examples to learn how different features are related to house prices.

Training Set = Data the model learns from.

2. Test Set#

The test set contains data that the model does not see during training. After training, we use it to evaluate how well the model performs on new, unseen examples. For example, with 1,000 observations, we might use:

Training Set: 800 observations (80%)
Test Set:     200 observations (20%)

The model learns from the 800 training examples and is evaluated on the 200 test examples.

Training = Learn

Testing = Evaluate on unseen data

But There Is a Problem…#

Suppose we train three models: Model A, Model B, and Model C. We evaluate all three on the test set and choose the best one.

Then we change some hyperparameters, check the test performance again, make another change, and check again.

What’s the problem?

We are now using the test set to make decisions about our model. Even though the model is not directly trained on the test data, our choices are being influenced by it. The test set is no longer truly “unseen.”

We need another dataset that we can use while developing and selecting our model.

This is the validation set.

3. Validation Set#

The validation set is used to evaluate different modeling choices during model development. The model does not learn from the validation data. Instead, we use the validation set to decide which model or model settings work best before touching the test set. For example, we may need to decide:

Which model should we use?
How complex should the model be?
Which hyperparameter values should we choose?

Quick Reminder: Parameters vs. Hyperparameters

Parameters are learned by the model from the training data. For example, the coefficients in linear regression are parameters.

Hyperparameters are settings that we choose to control the model or its complexity. Examples include the degree of polynomial regression, k in K-Nearest Neighbors, or the maximum depth of a decision tree.

So, the training set is used to learn the model’s parameters, while the validation set helps us choose the model and its hyperparameters. Instead of splitting our data into two parts, we now use three:

Figure: Training, validation, and test data. Left: The dataset is divided into training, validation, and test sets. Right: The training set is used to train the model, the validation set is used for model selection and tuning, and the test set is reserved for final evaluation on unseen data. Sources: LinkedIn and Medium.

For example, we might split the data as:

70% Training → 15% Validation → 15% Test

Note: The validation set is sometimes called a “holdout validation set” because we hold out a portion of the data for model selection and tuning.

Each set now has a different job:

Training Set: Learn the model’s parameters.
Validation Set: Select the model and tune its hyperparameters.
Test Set: Evaluate the final model on unseen data.

Example: Choosing Model Complexity#

Suppose we are comparing polynomial regression models with degrees 1, 4, and 15.

We first train all three models on the training set:

Training Set
     ↓
 ┌──────────┬──────────┬───────────┐
 Degree 1    Degree 4    Degree 15

Then we evaluate each model on the validation set:

Degree 1  → Validation MSE = 8.2
Degree 4  → Validation MSE = 2.1  ← Best
Degree 15 → Validation MSE = 7.5

Since Degree 4 has the lowest validation error, we select Degree 4.

Notice that we still have not used the test set. Only after we finish selecting and tuning the model do we evaluate the final model on the test set.

A simple way to remember this is to think about preparing for an exam:

Training Set → Study material
Validation Set → Practice exam
Test Set → Final exam

You learn from the study material, use the practice exam to make adjustments, and take the final exam only when you are ready.

Training vs. Validation vs. Test#

Dataset	Purpose	Does the Model Learn From It?
Training Set	Learn model parameters	Yes
Validation Set	Select models and tune hyperparameters	No
Test Set	Final evaluation	No

Important: The test set should remain untouched during model selection and hyperparameter tuning. It is used only at the end to estimate how well the final model generalizes to new data.

In practice, we can create the training and test sets using train_test_split() from scikit-learn.

import pandas as pd
from sklearn.model_selection import train_test_split

# Small example dataset
data = pd.DataFrame({
    "Size":     [1200, 1500, 1800, 2000, 2200, 2500, 2800, 3000, 3200, 3500],
    "Bedrooms": [2, 3, 3, 3, 4, 4, 4, 5, 5, 5],
    "Price":    [250, 300, 340, 380, 420, 460, 500, 550, 590, 640]
})

# X = input features, y = target
X = data[["Size", "Bedrooms"]]
y = data["Price"]

# Split: 80% training, 20% testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

print("Training observations:", len(X_train))
print("Testing observations:", len(X_test))

Training observations: 8
Testing observations: 2

Here, X contains the input features (Size and Bedrooms), while y contains the value we want to predict (Price).

train_test_split() randomly divides the data into two parts:

X_train, y_train → 80% of the data used for training
X_test, y_test → 20% of the data reserved for testing

Setting random_state=42 makes the random split reproducible, meaning we get the same train-test split each time we run the code.

# First, hold out 15% for the final test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42
)

# From the remaining 85%, hold out 15/85 for validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=15/85, random_state=42
)

print("Training observations:", len(X_train))
print("Validation observations:", len(X_val))
print("Testing observations:", len(X_test))

"""
This approach is called holdout validation because one fixed portion of the data is held out for validation.
"""

Training observations: 6
Validation observations: 2
Testing observations: 2

'\nThis approach is called holdout validation because one fixed portion of the data is held out for validation.\n'

Training vs. Validation Performance#

Now that we understand the role of the validation set, we can also use it to help identify overfitting.

Training and Validation Loss During Overfitting

Figure: Training and validation loss during model training. At first, both losses decrease as the model learns. After a certain point, training loss continues to decrease while validation loss begins to increase, indicating that the model is starting to overfit.

Key idea: A growing gap between training and validation performance can be a sign of overfitting.