Introduction: Model Evaluation and Validation

Imagine you are teaching a child to identify cats and dogs. If the child memorizes every single picture you show them, they may do great on those exact pictures. But when you show them a new cat they have never seen before, they might fail. On the other hand, if you teach them only one simple rule like “cats are always white,” they will also fail because many cats are not white.

Machine learning models behave in a very similar way. A good model should learn patterns, not memorize data or rely on overly simple assumptions. In this chapter, we’ll learn how to evaluate machine learning models so that they perform well not only on the training data but also on new, unseen data.

Overfitting and Underfitting

When training a machine learning model, our goal is not simply to achieve the highest accuracy on the training data. Instead, we want the model to perform well on data it has never seen before.

There are three possible situations.

Figure: Underfitting, Appropriate fitting, and Overfitting. Source: GeeksforGeeks

1. Underfitting

Underfitting happens when the model is too simple to capture the relationship in the data. Think of a first-grade student trying to solve a college-level math problem. No matter how hard they try, they simply don’t have enough knowledge.

Characteristics:

• Low training accuracy

• Low testing accuracy

• Fails to capture important patterns

Common Causes:

• Model is too simple

• Not enough training time

• Important features are missing

Example: predicting house price with the single rule “every house costs $300,000” ignores size, location, condition entirely. Both training and test performance are bad. That’s the tell.

How to Improve:

• Use a more complex model.

• Add more useful features.

• Train the model longer (if applicable).

2. Overfitting

Overfitting happens when the model memorizes the training data instead of learning general patterns. Imagine memorizing all questions from last year’s exam instead of understanding the concepts. If the exam changes slightly, you perform poorly.

Characteristics:

• Very high training accuracy

• Low testing accuracy

• Large gap between the two; this gap is the signature of overfitting

Common Causes:

• Model is too complex

• Too many features or parameters

• Too little training data

• Learning noise instead of the true pattern

Example: The model memorizes the exact prices of the houses it saw during training.

Training Data

House	Actual Price	Model Prediction
House A	$420K	$420K
House B	$513K	$513K
House C	$390K	$390K

The model gets 100% accuracy on the training data because it has memorized every example.

Test Data (new houses the model never saw)

House	Actual Price	Model Prediction
House D	$445K	$390K
House E	$375K	$513K

Although the model performs perfectly on the training data, it predicts the wrong price for House D and E because it memorized the training examples instead of learning how features such as house size, location, and number of bedrooms influence house prices.

Figure: Relationship between Bias, Variance, Underfitting, and Overfitting. Source: Towards Data Science

How to Reduce Overfitting:

• Use a simpler model.

• Collect more training data.

• Remove unnecessary features.

• Use regularization (covered in the “regression” chapter).

3. Good Fit

The ideal model lies between underfitting and overfitting. It learns meaningful patterns without memorizing every detail.

Characteristics:

• High training accuracy

• High testing accuracy

• Small gap between the two

	Training Accuracy	Testing Accuracy	Gap
Underfitting	Low	Low	Small (both bad)
Good Fit	High	High	Small (both good)
Overfitting	High	Low	Large

This table is the fastest way to diagnose which regime you’re in: look at both numbers, not just one.

Diagnosis rule of thumb: never look at training accuracy alone. 95% training / 93% test is a good fit. 95% training / 60% test is overfitting.

Visual Idea

Underfitting   → Too Simple
Good Fit       → Learns Patterns
Overfitting    → Memorizes Data