Introduction

A model making accurate predictions is not automatically overfitting, and a model with very high training accuracy is not automatically good. The distinction comes from generalization: does the performance hold on data the model did not train on, under an evaluation setup that was not contaminated by leakage? That is why the answer is found in validation design, not in one impressive metric alone.

Compare Training and Validation Behavior

Overfitting usually reveals itself as a widening gap between training performance and validation performance. Good prediction means the model performs well on held-out data too.

1from sklearn.datasets import make_classification
2from sklearn.model_selection import train_test_split
3from sklearn.tree import DecisionTreeClassifier
4
5X, y = make_classification(random_state=0)
6X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
7
8model = DecisionTreeClassifier(random_state=0)
9model.fit(X_train, y_train)
10
11print("train:", model.score(X_train, y_train))
12print("test:", model.score(X_test, y_test))

If training score is near perfect but test score drops sharply, that is a classic sign of overfitting.

Use a Proper Validation Strategy

A single random split can be misleading, especially with small datasets. Cross-validation gives a stronger picture of whether the model's performance is stable.

1from sklearn.model_selection import cross_val_score
2from sklearn.ensemble import RandomForestClassifier
3
4model = RandomForestClassifier(random_state=0)
5scores = cross_val_score(model, X, y, cv=5)
6print(scores)
7print(scores.mean())

A genuinely good predictor should look reasonably strong across multiple folds, not only on one lucky split.

Watch for Data Leakage

A model can appear excellent while still being untrustworthy if information leaked from the future, the target, or the test set into training. Leakage often looks like good prediction when it is really evaluation contamination.

That is why distinguishing overfitting from genuine predictive power is not only about regularization or model complexity. It is also about whether the experimental setup was honest.

If preprocessing, scaling, feature selection, or imputation was performed using the full dataset before splitting, the measured quality may be inflated even if the train and validation numbers look close.

Learning Curves Help Diagnose the Pattern

Learning curves show how training and validation performance change as more data is used. They are useful because they separate three cases: underfitting, overfitting, and healthy generalization.

If the training score stays much higher than the validation score, you likely have overfitting. If both are low, the model may be too weak. If both are high and close together, that is what "good prediction" is supposed to look like.

Good Prediction Is About Stability, Not Only Peak Accuracy

One strong test score is encouraging, but it is not the whole story. Good predictive models remain reasonably accurate across folds, time periods, and realistic deployment conditions.

That means calibration, robustness to distribution shift, and repeatability matter too. A model that gets one very high number on one dataset split may still be fragile.

The strongest sign of real predictive value is consistent performance under evaluation designs that mimic how the model will actually be used.

Prefer Simpler Explanations First

If a model suddenly performs suspiciously well, check simpler explanations before celebrating. Leakage, target duplication, duplicated rows, and split mistakes are more common than miraculous performance.

That discipline is how practitioners keep themselves from confusing accidental advantage with real signal.

Common Pitfalls

• Judging the model only by training accuracy.
• Treating one random train-test split as conclusive proof of generalization.
• Ignoring leakage in preprocessing or feature construction.
• Equating a complex model with overfitting without checking validation behavior.
• Celebrating unusually high scores before validating whether the evaluation setup is realistic.

Summary

• Overfitting is about poor generalization, not just high training accuracy.
• Compare training results with honest validation or test results.
• Use cross-validation when one split is too fragile.
• Check for leakage before trusting great metrics.
• Good prediction is stable out-of-sample performance, not a single impressive number.