Understanding F1-score in Multi-class Classification

In the realm of multi-class classification, evaluating model performance requires reliable and interpretable metrics. Among these, the F1-score is a popular choice, especially when dealing with imbalanced data. It combines precision and recall into a single metric, providing a harmonic mean that is sensitive to both false positives and false negatives.

Technical Explanation

Before diving into F1-score computations for multi-class settings, it is essential to clarify the primary components:

• Precision: The ratio of true positive predictions to the total predicted positives. It indicates how many of the items predicted as a certain class are truly of that class.
  $$\text{Precision} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Positives (FP)}}$$
• Recall (Sensitivity): The ratio of true positive predictions to the actual positives. It measures how well the model can identify all relevant instances.
  $$\text{Recall} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}$$
• F1-score: The harmonic mean of precision and recall, which balances the trade-off between these two metrics.
  $$\text{F1-score} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

In a binary classification scenario, calculating these metrics is straightforward. However, for multi-class classification, this process must be adapted to account for multiple classes.

F1-score Computation for Multi-class Classification

In a multi-class setting, classes are often independently assessed using a "one-vs-all" approach. Here's how F1-scores are calculated for each class:

1. One-vs-All Precision and Recall: For each class, treat that class as the positive class and all other classes as the negative class to compute precision and recall.
2. Per Class F1-score: Calculate the F1-score for each class using the precision and recall values derived in the previous step.
3. Macro and Weighted F1-scores:
   • Macro F1-score: Calculate the mean F1-score across all classes, treating all classes equally regardless of their size.
     $$\text{Macro F1-score} = \frac{1}{N}\sum_{i=1}^{N} F1_i$$
     where $F1_i$ is the F1-score for class $i$, and $N$ is the number of classes.
   • Weighted F1-score: Calculate the average of F1-scores, weighted by the number of true instances for each class.
     $$\text{Weighted F1-score} = \frac{1}{\sum_{i=1}^{N} w_i} \sum_{i=1}^{N}w_i \cdot F1_i$$
     where $w_i$ is the number of true instances of class $i$.

Example

Consider a multi-class classification problem with three classes: A, B, and C. Here's a confusion matrix showing the prediction results:

Based on this matrix, the F1-score for each class can be computed:

• Class A:
  • Precision = 35 / (35 + 3 + 4) = 0.833
  • Recall = 35 / (35 + 5 + 10) = 0.700
  • F1-score = 2 * 0.833 * 0.700 / (0.833 + 0.700) = 0.761
• Class B:
  • Precision = 40 / (5 + 40 + 8) = 0.769
  • Recall = 40 / (3 + 40 + 7) = 0.800
  • F1-score = 2 * 0.769 * 0.800 / (0.769 + 0.800) = 0.784
• Class C:
  • Precision = 38 / (10 + 7 + 38) = 0.679
  • Recall = 38 / (4 + 8 + 38) = 0.760
  • F1-score = 2 * 0.679 * 0.760 / (0.679 + 0.760) = 0.717

Summary Table

By averaging these scores, we can compute macro and weighted F1-scores, considering the specific context and distribution of classes.

Conclusion

The F1-score per class in multi-class classification provides valuable insights into how well a model performs across different categories. When facing imbalanced datasets, these class-specific metrics become even more crucial, ensuring that each class's evaluation accounts for its true proportion and significance. Determining when to use a macro versus a weighted F1-score depends directly on the classification task's requirements and the relative importance of each class.