Understanding the reasons behind a high Area Under the Curve (AUC) and low accuracy in the context of a balanced dataset can seem perplexing at first. However, these metrics, while related, capture different aspects of a model's performance. To address these nuances, we need to explore the nature of AUC, accuracy, and the potential factors influencing their metrics.

AUC vs. Accuracy

Accuracy

Accuracy is a straightforward metric: it is the ratio of correctly predicted observations to the total observations. In a perfectly balanced dataset, one might expect both accuracy and AUC to align closely since class distributions are even. However, this isn't always the case.

Mathematically, accuracy is represented as follows:

$$\text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Population}}$$

AUC

AUC, which stands for Area Under the Receiver Operating Characteristic (ROC) Curve, provides a measure of a model's ability to distinguish between classes. It summarizes the trade-off between true positive rate (sensitivity) and false positive rate (1-specificity) across various threshold levels.

AUC is often more insightful in capturing the discriminatory power of models especially with imbalanced classes, but it can be revealing even in balanced settings. The formula for calculating AUC isn't as straightforward, as it involves integrating the ROC curve.

Reasons for High AUC and Low Accuracy

The disparity arises primarily because of how these metrics are fundamentally designed to measure different characteristics of model performance.

1. Threshold Sensitivity: • Accuracy operates on a fixed threshold (often 0.5 in binary classification), which may not be optimal for prediction tasks. • AUC evaluates performance across all potential thresholds, giving a more holistic view of the model's performance.
2. Class Distributions at Specific Thresholds: • A model might predict probabilities well, capturing class distinctions vividly, which results in high AUC. • However, if predictions hover near threshold boundaries, slight changes can switch classes, impacting accuracy.
3. Misclassification in Balanced Sets: • Despite balanced datasets, there could be clusters of instances that are consistently misclassified due to feature overlap, leading to lower accuracy. • AUC might remain unaffected as long as relative ranking of predictions is preserved.
4. Bias vs. Variance Error: • Models with high variance might perfectly separate classes most of the time (high AUC) but fail at specific data points, reducing accuracy. • Conversely, systematic biases at the threshold level could skew accuracy without affecting AUC.

Examples and Scenarios

Consider a binary classification task on a balanced dataset where:

• The probability estimates for true positive examples cluster between 0.6 and 0.7. • The probability estimates for true negative examples cluster between 0.4 and 0.5.

Here, the separation is broad enough to give a high AUC, but the threshold of 0.5 might misclassify several instances, lowering accuracy.

Implications and Considerations

• Threshold Adjustment: • Consider varying thresholds to identify an optimal point balancing sensitivity and specificity.

• Use of AUC vs. Accuracy: • Rely on AUC for insight into ranking and classification performance. • Use accuracy when model output is meant for final decision making with specific cost implications on errors.

Conclusion

The discord between high AUC and low accuracy, even in balanced datasets, underscores the importance of thoughtful metric selection and cross-evaluation. While AUC provides a glimpse into how well a model captures inherent class separability, accuracy offers a more direct measure of performance at a chosen decision threshold. Understanding and interpreting these metrics in tandem rather than isolation can lead to better model evaluation and task-specific adjustments.