Limitation of Holdout Validation

Limitation of Holdout Validation#

Holdout validation is simple, but it has two important limitations.

First, if the dataset is small, setting aside a validation set leaves us with less data for training.
Second, the evaluation can depend on which observations happen to be placed in the validation set.

For example, suppose we have only 100 observations and use 20 for validation. If those 20 observations happen to be unusually easy or unusually difficult, the validation score may not represent the model’s true performance very well. There is also no single train-validation-test split that works for every problem. The appropriate split depends on factors such as the size of the dataset and the task.

So instead of relying on one validation split, what if we could evaluate the model using several different validation splits? This leads us to cross-validation.

Cross-Validation#

Cross-validation evaluates a model using multiple train-validation splits instead of relying on a single validation set.

The general idea is:

Split the available training data into several parts.
Train the model on some parts.
Validate it on another part.
Repeat the process using different parts for validation.
Combine, usually by averaging, the validation scores.

This gives us a more reliable estimate of how the model performs across different subsets of the data.

Cross-validation asks: Does the model perform well across different subsets of the data, rather than just one particular validation split?

Notice that we still keep the test set separate.

Full Dataset
     │
     ├── Test Set → Keep untouched until the end
     │
     └── Training Data
              │
              └── Cross-Validation → Model selection/tuning

Once we have selected our model using cross-validation, we can train the chosen model on the available training data and evaluate it once on the test set.

Figure: K-Fold Cross-Validation. In each round, one fold is held out for evaluation while the remaining folds are used for training. Source: Towards Data Science.

Important Note: In cross-validation, the held-out fold is sometimes called the “test fold” or “test data.” In our train-validation-test terminology, it serves as the validation fold for model selection and tuning. The final test set is completely separate and is used only after model selection and tuning are complete to evaluate the final model on unseen data.

1. K-Fold Cross-Validation#

One of the most commonly used cross-validation techniques is K-Fold Cross-Validation. In K-Fold Cross-Validation, we divide the training data into K approximately equal-sized groups, called folds.

Suppose K = 5:

Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5

We train and validate the model five times. Each time, one fold is used as the validation set, while the remaining four folds are used for training.

Round	Training Folds	Validation Fold
1	2, 3, 4, 5	1
2	1, 3, 4, 5	2
3	1, 2, 4, 5	3
4	1, 2, 3, 5	4
5	1, 2, 3, 4	5

Notice that every observation is used for validation exactly once and for training in the other K - 1 rounds.

Suppose the five validation accuracies are:

Fold 1: 88%
Fold 2: 91%
Fold 3: 89%
Fold 4: 90%
Fold 5: 92%

We average these scores:

Average CV Accuracy = 90%

Rather than judging the model based on one validation split, we now estimate its performance using five different validation folds.

K-Fold Cross-Validation with Separate Test Set

Figure: K-Fold Cross-Validation with a separate test set. The training set is divided into K folds, with each fold taking a turn as the validation set. Cross-validation is used for model selection and tuning, while the test set remains separate and untouched until the final model evaluation. Source: Aptech.

Choosing K#

Two common choices are:

K = 5
K = 10

A larger K means that more data is used for training in each round, but the model must also be trained more times, increasing computational cost.

Important: The folds used during cross-validation come from the training data. They are used for training and validation during model selection and tuning. The final test set is not one of these folds and remains untouched until the final evaluation.

K-Fold in Scikit-Learn#

from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

kfold = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(
    model,
    X_train,
    y_train,
    cv=kfold,
    scoring="accuracy"
)

print("Fold Scores:", scores)
print("Average CV Accuracy:", scores.mean())

Here, the model is trained and validated five times, and scores.mean() gives the average validation score.

2. Stratified K-Fold Cross-Validation#

Regular K-Fold divides observations into folds, but there is another issue we need to consider for classification problems. Suppose we have 100 patients:

90 → No Disease
10 → Disease

This is an imbalanced dataset because one class is much more common than the other. With regular K-Fold, the class distribution can vary across folds. One fold might contain many Disease cases, while another might contain very few. This can make our evaluation less reliable.

Stratified K-Fold addresses this problem by approximately preserving the class proportions in each fold. For example:

Original Dataset:
90% No Disease
10% Disease

        ↓

Fold 1: ≈ 90% No Disease | ≈ 10% Disease
Fold 2: ≈ 90% No Disease | ≈ 10% Disease
Fold 3: ≈ 90% No Disease | ≈ 10% Disease
...

Figure: Stratified K-Fold Cross-Validation. The class distribution of the original dataset is approximately preserved in each fold, making evaluation more reliable for imbalanced classification datasets. Source: Dataaspirant.

A Simple Example: K-Fold vs. Stratified K-Fold#

Suppose we have 12 observations from three classes, and the observations happen to be ordered by class:

A A A A A A | B B B B | C C

The overall class distribution is:

Class A: 6 observations (50%)
Class B: 4 observations (33%)
Class C: 2 observations (17%)

Now suppose we use 2-fold cross-validation.

With regular K-Fold without shuffling, the observations are divided according to their existing order:

Method	Fold 1	Fold 2
K-Fold (without shuffling)	A, A, A, A, A, A	B, B, B, B, C, C
Stratified K-Fold	A, A, A, B, B, C	A, A, A, B, B, C

Regular K-Fold produces very different class distributions: Fold 1 contains only Class A, while Fold 2 contains only Classes B and C.

With Stratified K-Fold, each fold contains:

3 A's + 2 B's + 1 C

so each fold preserves the original class proportions of approximately 50% A, 33% B, and 17% C.

Key idea: Regular K-Fold does not guarantee similar class proportions across folds. Stratified K-Fold explicitly tries to preserve the class distribution in each fold.

Note: Shuffling before regular K-Fold can reduce problems caused by ordered data, but it still does not guarantee similar class proportions. Stratified K-Fold is specifically designed for this purpose.

Stratified K-Fold is especially useful for classification problems, particularly when the classes are imbalanced.

Stratified K-Fold in Scikit-Learn#

import pandas as pd
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression

# Small classification dataset
data = pd.DataFrame({
    "Age":    [22, 25, 28, 32, 35, 38, 42, 45, 48, 52, 55, 58],
    "Income": [35, 40, 42, 48, 52, 55, 60, 62, 68, 72, 75, 80],
    "Class":  ["A", "A", "A", "A", "A", "A",
               "B", "B", "B", "B",
               "C", "C"]
})

# X = input features, y = class label
X = data[["Age", "Income"]]
y = data["Class"]

# Classification model
model = LogisticRegression(max_iter=1000)

# 2-fold Stratified Cross-Validation
skfold = StratifiedKFold(
    n_splits=2,
    shuffle=True,
    random_state=42
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=skfold,
    scoring="accuracy"
)

print("Fold Scores:", scores)
print("Average CV Accuracy:", scores.mean())

3. Leave-One-Out Cross-Validation (LOOCV)#

What if we take the K-Fold idea to the extreme?

Suppose our dataset contains N observations. Instead of choosing K = 5 or K = 10, we choose:

K = N

Figure: Leave-One-Out Cross-Validation (LOOCV). Left: In each iteration, one observation is held out for validation while all remaining observations are used for training. Right: Each observation takes a turn as the validation observation, producing N validation scores that are combined to estimate overall model performance. Sources: Dataaspirant and KDnuggets.

This is called Leave-One-Out Cross-Validation (LOOCV).

In each round:

N - 1 observations are used for training.
1 observation is used for validation.

For example, suppose we have only five observations: A, B, C, D, and E.

Round	Training	Validation
1	B, C, D, E	A
2	A, C, D, E	B
3	A, B, D, E	C
4	A, B, C, E	D
5	A, B, C, D	E

Every observation gets exactly one turn as the validation observation.

Advantages#

Almost all available data is used for training in every round.
Can be useful when the dataset is very small.

Disadvantages#

Requires training the model N times.
Can be computationally expensive for large datasets.
The resulting estimate can have higher variance than K-Fold with moderate values such as 5 or 10.

For these reasons, 5-Fold or 10-Fold Cross-Validation is often preferred in practice.

LOOCV in Scikit-Learn#

from sklearn.model_selection import LeaveOneOut, cross_val_score

loo = LeaveOneOut()

scores = cross_val_score(
    model,
    X_train,
    y_train,
    cv=loo,
    scoring="accuracy"
)

print("Average LOOCV Accuracy:", scores.mean())

Comparing Cross-Validation Techniques#

Method	How It Works	When Is It Useful?
K-Fold	Divides data into K folds and rotates the validation fold	General-purpose cross-validation
Stratified K-Fold	K folds while approximately preserving class proportions	Classification, especially with imbalanced classes
LOOCV	Leaves one observation out for validation each round	Very small datasets

The main idea behind all three techniques is the same: do not judge a model based on just one validation split. By evaluating it across different subsets of the data, we get a more reliable picture of how well it is likely to generalize.