Introduction

When evaluating regression models, two critical metrics often used are the Mean Absolute Error (MAE) and the coefficient of determination, commonly represented as $R^2$. The MAE provides a straightforward measure of the average magnitude of errors in a set of predictions, without considering their direction, while $R^2$ offers insights into the proportion of variance in the dependent variable that is predictable from the independent variables. Under certain conditions, a model might yield both a low MAE and a low $R^2$ score simultaneously. This scenario might be initially puzzling as one metric suggests high accuracy (low MAE), while the other implies poor fit (low $R^2$). Understanding why this happens requires a deeper dive into how each metric functions and their limitations.

Understanding MAE and $R^2$

Mean Absolute Error (MAE)

The Mean Absolute Error is calculated using the formula:

$$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

where $y_i$ is the actual value, $\hat{y}_i$ is the predicted value, and $n$ is the number of observations. MAE reflects the average absolute difference between observed and predicted values, providing a clear measure of prediction accuracy.

Coefficient of Determination ($R^2$)

$R^2$ is computed as:

$$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}$$

where $\text{SS}_{\text{res}}$ is the sum of squares of residuals (i.e., $\sum{(y_i - \hat{y}_i)^2}$) and $\text{SS}_{\text{tot}}$ is the total sum of squares (i.e., $\sum{(y_i - \bar{y})^2}$). The $R^2$ metric essentially measures how well the data fits the model, with values closer to 1 indicating a better fit.

Potential Scenarios of Low MAE and Low $R^2$

Presence of Data Variability

• Uniformly Distributed Targets: • If the target variable exhibits minimal variation (i.e., low variance), even a model predicting values close to the central tendency (such as the mean) can result in a low MAE. However, if the model lacks the ability to explain the variability, it might also show a low $R^2$.

Insensitivity to Scale

• Difference in Scale Sensitivity: • A model with predictions that precisely follow the scale of the dependent variable (e.g., small numerical range) might incur a low average error (low MAE). However, if the inherent variability or scale cannot be captured (potentially due to noise or insufficient variability in inputs), the $R^2$ can remain low.

Influence of Outliers

• Presence of Outliers: • Outliers strongly influence $R^2$ but have a muted effect on MAE. If your dataset contains significant outliers, they might drastically lower $R^2$ even if the model appropriately predicts the non-outlier data, leading to a reasonable MAE.

Non-linear Relationships

• Non-linear Patterns: • A linear model applied to data with non-linear patterns might capture a central tendency of predictions leading to a low MAE, yet explain only a small amount of variance, hence displaying a low $R^2$.

Model Complexity

• Underfitting: • An overly simplistic model can result in predictions that approximate an average with minimal prediction error. Although such prediction errors might be minimal (low MAE), the model might still fail to capture the specific variations in the data leading to a low $R^2$.

Key Points Summary

Conclusion

Understanding regression model metrics such as MAE and $R^2$ is essential to interpreting model performance. While low errors might denote accurate predictions in terms of magnitude, explaining data variability is a separate concern influenced by factors such as data distribution, model choice, and inherent data noise. Recognizing the environments and conditions that lead to divergent results between these metrics aids in thorough model evaluation and ensuing improvements. Balancing both error minimization and variance explanation ultimately leads to better-performing models and insightful data understanding.