Bias, Variance, and the Bias-Variance Tradeoff

To better understand underfitting and overfitting, we introduce two important concepts: bias and variance.

Remember, underfitting and overfitting are the symptoms. Bias and variance are the underlying causes.

• Underfitting → Model is too simple → High Bias

• Overfitting → Model is too complex → High Variance

1. Bias

Bias measures how wrong a model’s assumptions are in a systematic sense. Imagine training the same type of model on many different training datasets drawn from the same population. Bias is the average difference between the predictions and the true values across all of those trained models.

High bias means the model is too simple to capture the true relationship in the data. As a result, it makes similar mistakes regardless of which training dataset it was trained on, leading to consistent errors. High bias usually causes:

• Underfitting

• Missing important relationships in the data

Example: Trying to fit a straight line through data that clearly follows a curve. No matter how you rotate or shift the line, it cannot bend to match the underlying pattern.

Figure: Graphical representation of bias. Left: High bias: the model is too simple to capture the true relationship, resulting in underfitting. Right: Low bias: the model is flexible enough to closely fit the training data; however, if this is accompanied by high variance, the model may overfit. Source: TutorialsPoint.

2. Variance

Variance measures how much a model’s predictions change when it is trained on different training datasets. Imagine training the same type of model on several slightly different training datasets. If the model produces very different predictions each time, it has high variance.

In other words, small changes in the training data can lead to large changes in the model’s predictions.

A model with high variance is too sensitive to the training data. Instead of learning the true underlying pattern, it may also learn random noise or small fluctuations in the data. As a result, the model performs very well on the training data but does not generalize well to new, unseen data. High variance usually leads to:

• Overfitting

• Poor performance on unseen data, because the model learns the noise in the training data instead of the true underlying pattern.

Note: In statistics, variance measures the spread of numbers. In machine learning, variance measures how sensitive a model is to changes in the training data.

Figure: Illustration of model variance. Left: High variance: the green point represents the average prediction, while the red points show predictions from models trained on different datasets. Their wide spread indicates high variance and a greater tendency to overfit. Right: Low variance: the red points are closely clustered around the green point, indicating more consistent predictions and lower sensitivity to changes in the training data. Source: TutorialsPoint.

Understanding with an Example

Suppose we train three models on the same data:

Model	Behavior	Bias	Variance
Model A	Straight line; misses the pattern	High	Low
Model B	Smooth curve; captures the trend	Balanced	Balanced
Model C	Very complex curve; follows nearly every training point	Low	High

Figure: Relationship between model complexity, bias, and variance. Left: An underfitted model is too simple and has high bias. Middle: A good fit captures the underlying pattern with a good balance between bias and variance. Right: An overfitted model is too complex and has high variance, following noise in the training data. Source: Miro Medium.

Another way to visualize bias and variance is to imagine a model’s predictions as shots aimed at a bullseye:

Figure: Visualizing bias and variance using a bullseye. The center represents the true value, while the dots represent model predictions across different training datasets. Low bias means predictions are centered near the true value, while low variance means predictions are tightly clustered. Ideally, we want both low bias and low variance. Source: Cornell CS4780.

Bias-Variance Tradeoff

Bias and variance are closely related to the complexity of a model. One of the biggest challenges in machine learning is finding the right balance between them. This is known as the bias-variance tradeoff.

Why is there a tradeoff? As model complexity increases, bias tends to decrease (because the model can capture more complex patterns), while variance tends to increase (because the model becomes more sensitive to the training data). If we decrease model complexity, the opposite happens: bias increases while variance decreases.

Therefore:

• A simple model usually has ↑ bias and ↓ variance → more likely to underfit.

• A complex model usually has ↓ bias and ↑ variance → more likely to overfit.

Figure: Bias-variance tradeoff. The x-axis represents model complexity, and the y-axis represents prediction error. As model complexity increases, bias decreases while variance increases. A model that is too simple tends to underfit, while a model that is too complex tends to overfit. The optimal region represents the "sweet spot" where bias and variance are balanced and total prediction error is minimized. Source: TutorialsPoint.

The goal is to find the “sweet spot” that balances bias and variance and minimizes prediction error on unseen data.

Total Error

The prediction error of a machine learning model can be thought of as three components:

Total Error ≈ Bias² + Variance + Irreducible Error

where:

• Bias²: Error caused by the model’s overly simple assumptions.

• Variance: Error caused by the model being too sensitive to the training data.

• Irreducible Error: Random noise or uncertainty in the data that no model can completely eliminate.

The goal is to choose a model complexity that balances bias and variance, resulting in the lowest possible total error on unseen data.

Key idea: We cannot completely eliminate irreducible error, but we can control bias and variance by choosing an appropriate model and its complexity.

Techniques to Balance Bias and Variance

In practice, we can adjust the model and training process to find a better balance between bias and variance.

To reduce high bias (underfitting):

• Use a more complex model.

• Add useful features that capture important patterns.

• Reduce regularization, if the model is overly constrained.

• Train longer, when applicable.

To reduce high variance (overfitting):

• Use a simpler model.

• Collect more training data, when possible.

• Remove unnecessary or noisy features.

• Use regularization to prevent the model from becoming too complex.

But how do we know which model complexity gives the best balance? We need to evaluate different models on data they were not trained on. This leads us to training, validation, and test datasets and cross-validation.

degree_range = range(1, 16)
train_errors = []
test_errors = []

for d in degree_range:
    model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(X_train, y_train)
    train_errors.append(mean_squared_error(y_train, model.predict(X_train)))
    test_errors.append(mean_squared_error(y_test, model.predict(X_test)))

plt.figure(figsize=(8, 5))
plt.plot(degree_range, train_errors, marker="o", label="Training error", color="steelblue")
plt.plot(degree_range, test_errors, marker="o", label="Test error", color="darkorange")
plt.axvline(x=4, color="gray", linestyle="--", alpha=0.6, label="Sweet spot (approx.)")
plt.xlabel("Model complexity (polynomial degree)")
plt.ylabel("Mean Squared Error")
plt.title("Validation curve: the bias-variance tradeoff in action")
plt.legend()
plt.show()

This plot is the bias-variance tradeoff:

• On the left (low degree): both errors are high → high bias, underfitting.

• In the middle: both errors are low and close → balanced bias/variance, good fit.

• On the right (high degree): training error keeps falling toward zero, but test error climbs back up → high variance, overfitting.

This chart is called a validation curve, and it’s the standard tool for picking model complexity (polynomial degree, tree depth, number of neural net layers, regularization strength, etc.) in practice.