Machine Learning Evaluation Metrics

Once we have trained and validated a model, we still need to answer another question:

What does “good performance” actually mean?

There is no single evaluation metric that is best for every machine learning problem.

For classification, common metrics include:

• Accuracy

• Confusion Matrix

• Precision

• Recall

• F1 Score

The appropriate metric depends on the problem and, importantly, on which types of mistakes matter most.

1. Accuracy

Accuracy measures the proportion of predictions that the model gets correct.

Accuracy = Number of Correct Predictions / Total Number of Predictions

Suppose a model makes 100 predictions and gets 90 correct:

Accuracy = 90 / 100 = 0.90 = 90%

Accuracy is simple and intuitive, but it can be misleading when the classes are imbalanced.

Imbalanced data means that one class has many more observations than another class.

Example

Suppose we have 100 patients:

95 → No Disease
 5 → Disease

Imagine a model simply predicts “No Disease” for everyone. It gets:

95 / 100 = 95% accuracy

That sounds excellent. But the model detected zero of the five patients who actually had the disease. So accuracy alone does not always tell the full story.

2. Confusion Matrix

A confusion matrix shows not only how many predictions were correct, but also what types of mistakes the model made. For binary classification:

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

Figure: Confusion matrix and common classification metrics. The four main outcomes are True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). These values are used to calculate metrics such as accuracy, precision, sensitivity (recall), and specificity. Source: Encord.

Understanding the Four Cases

True Positive (TP):
Actual = Positive, Prediction = Positive → Correct

True Negative (TN):
Actual = Negative, Prediction = Negative → Correct

False Positive (FP):
Actual = Negative, Prediction = Positive → Incorrect

False Negative (FN):
Actual = Positive, Prediction = Negative → Incorrect

A useful way to remember the terminology:

True/False tells us whether the prediction was correct.
Positive/Negative tells us what the model predicted.

3. Precision

Precision asks:

Of all the cases the model predicted as positive, how many were actually positive?

Precision = TP / (TP + FP)

High precision means the model produces few false positives.

Example

Suppose an email spam detector labels 20 emails as spam, but only 15 are actually spam:

Precision = 15 / 20 = 75%

Precision is important when false positives are costly.

For example, we may not want an important legitimate email incorrectly sent to the spam folder.

4. Recall

Recall asks:

Of all the actual positive cases, how many did the model successfully find?

Recall = TP / (TP + FN)

High recall means the model produces few false negatives.

Example

Suppose 20 fraudulent transactions actually occurred, but the model detected only 16:

Recall = 16 / 20 = 80%

Recall is important when missing a positive case is costly.

Precision vs. Recall

A simple way to remember the difference:

Precision → When the model says POSITIVE, how often is it right?

Recall    → Of all the REAL positives, how many did the model find?

Which one matters more depends on the problem.

For a spam filter, false positives may be especially annoying because legitimate emails could be hidden.

For fraud detection, missing fraudulent transactions may be particularly costly, so recall may deserve more attention.

5. F1 Score

Sometimes we care about both precision and recall. The F1 Score combines them into a single metric using their harmonic mean:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

The F1 score is high only when both precision and recall are reasonably high. For example:

Precision = 0.80
Recall    = 0.90

F1 ≈ 0.85

F1 score is particularly useful when:

• Classes are imbalanced.

• Both false positives and false negatives matter.

• We want one metric that summarizes precision and recall.

Putting the Metrics Together

Metric	Main Question
Accuracy	How many predictions were correct overall?
Precision	When the model predicts positive, how often is it correct?
Recall	How many of the actual positives did the model find?
F1 Score	How well does the model balance precision and recall?

Key idea: There is no universally “best” evaluation metric. The correct metric depends on the problem and the consequences of different types of errors.

Confidence and Predicted Probabilities

Many classification models do more than simply predict a class such as:

Spam
Not Spam

They can also produce a predicted probability for each class. For example:

P(Spam)     = 0.92
P(Not Spam) = 0.08

The model would predict Spam because it assigns a higher probability to that class. Informally, we may describe the 0.92 probability as the model being highly confident in its prediction.

However, confidence should be interpreted carefully:

High confidence does not guarantee that the prediction is correct.

A model can be confident and wrong. For example:

Model prediction: Cat
Predicted probability: 98%

Actual class: Dog

The model was highly confident, but its prediction was still incorrect.

Ideally, predicted probabilities should also be well calibrated. For example, among many predictions made with approximately 80% probability, we would expect roughly 80% to be correct.

Therefore, when evaluating a classifier, we may want to consider not only which class it predicts, but also the probabilities associated with those predictions.

Log Loss: When Confidence Matters

How can we evaluate the quality of these predicted probabilities? One commonly used metric is Log Loss.

Log Loss evaluates a model’s predicted probabilities and gives a larger penalty to predictions that are confident but incorrect. In other words, it considers not only whether a prediction is correct or incorrect, but also the probability the model assigned to the actual outcome.

For binary classification:

\[ \text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i\log(p_i)+(1-y_i)\log(1-p_i) \right] \]

where:

• \(N\) = number of observations

• \(y_i\) = actual class (0 or 1)

• \(p_i\) = predicted probability of class 1

An Intuitive Example

Suppose the true class is Disease (1):

Model	Predicted Probability of Disease	Prediction	Result
Model A	0.90	Disease	Correct and confident
Model B	0.55	Disease	Correct, but less confident
Model C	0.10	No Disease	Wrong and confident

Both Model A and Model B make the correct class prediction, but Model A assigns a higher probability to the correct class.

Model C is different: it assigns only a 10% probability to the true class, meaning it is highly confident in the wrong answer. Therefore, it receives a much larger Log Loss penalty.

Key idea: Log Loss strongly penalizes confident incorrect predictions and favors models that assign high probabilities to the correct outcomes.

For Log Loss, lower is better:

Lower Log Loss  → Better
Log Loss near 0 → Very good
Higher Log Loss → Worse

Unlike accuracy, which looks only at the final class prediction, Log Loss uses the predicted probabilities themselves. This allows two models with the same accuracy to receive different Log Loss values if the quality of their probability predictions is different.

---

Note: Log Loss can also be extended to multiclass classification, where the predicted probabilities across all classes are considered. Here, we focus on the binary case for simplicity.

For multiclass classification:

\[ \text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{M} x_{ij}\log(p_{ij}) \]

where:

• \(N\) = number of observations

• \(M\) = number of classes

• \(x_{ij}\) = 1 if observation \(i\) belongs to class \(j\), otherwise 0

• \(p_{ij}\) = predicted probability that observation \(i\) belongs to class \(j\)