Summary of the Chapter

In this chapter, we learned how to evaluate whether a machine learning model can generalize to new, unseen data. We explored underfitting and overfitting, how they relate to bias, variance, and model complexity, and how the bias-variance tradeoff helps us think about choosing an appropriate model.

We also learned how training, validation, and test sets play different roles in model development, and how cross-validation provides a more reliable approach to model selection and evaluation. Finally, we introduced common classification evaluation measures, including accuracy, precision, recall, F1 score, confusion matrices, and predicted probabilities.

The main lesson is simple: a good machine learning model is not the one that performs best on data it has already seen—it is the one that performs well on new, unseen data.

Quick Reference

Concept	Main Idea
Underfitting	Model is too simple and fails to capture important patterns.
Overfitting	Model fits the training data too closely and generalizes poorly.
Bias	Systematic error caused by overly simple model assumptions.
Variance	Sensitivity of the model to changes in the training data.
Bias-Variance Tradeoff	Balance bias and variance to improve generalization.
Training Set	Used to learn model parameters.
Validation Set	Used for model selection and hyperparameter tuning.
Test Set	Used for final evaluation on unseen data.
Cross-Validation	Evaluates a model across multiple train-validation splits.
K-Fold	Divides training data into K folds and rotates the validation fold.
Stratified K-Fold	Approximately preserves class proportions across folds.
LOOCV	Uses one observation for validation and the rest for training in each round.

Key Takeaways

• A good model should learn patterns and generalize, not simply memorize training data.

• A model that is too simple may underfit, while a model that is too complex may overfit.

• Keep the test set separate and untouched until the final evaluation.

• Use validation data or cross-validation for model selection and hyperparameter tuning.

• Choose the appropriate cross-validation method based on the problem and data.

• Do not rely on accuracy alone, especially for imbalanced classification problems.

Important Reminders

• Do not overuse the validation set. We use validation results to choose models and tune hyperparameters. If we keep changing the model based on the same validation set, our choices may become too specific to that validation data. This is why we keep a separate test set for the final evaluation.

• Avoid data leakage. As discussed in the Feature Engineering chapter, information from the validation or test set should not be used to help train the model. Also avoid putting duplicate or very similar observations in both the training and evaluation sets, because the model may have already seen almost the same examples.

Key idea: The test set should represent truly unseen data so that we can fairly evaluate how well the model will perform on new examples.