Introduction

A confusion matrix is one of the most useful ways to inspect a classifier, because it shows exactly where predictions land: true positives, false positives, true negatives, and false negatives. The catch in scikit-learn is that cross-validation APIs usually expect scalar scores, while a confusion matrix is a table.

Why a Confusion Matrix Does Not Fit cross_val_score

The scoring argument in cross_val_score is designed for values that can be compared and averaged across folds. Metrics such as accuracy, recall, and ROC AUC work well because each fold produces one number. A confusion matrix produces multiple counts, so there is no obvious built-in way to average it inside cross_val_score.

That means this does not do what most people want:

1from sklearn.metrics import confusion_matrix
2from sklearn.model_selection import cross_val_score
3from sklearn.linear_model import LogisticRegression
4
5model = LogisticRegression(max_iter=1000)
6
7# This is not a valid scorer for cross_val_score.
8scores = cross_val_score(model, X, y, scoring=confusion_matrix, cv=5)

confusion_matrix expects y_true and y_pred, not an estimator, features, and labels. More importantly, it returns an array rather than a single score.

The Practical Options

You usually want one of three patterns.

The first pattern is to score with a scalar derived from the confusion matrix. For example, precision, recall, specificity, balanced accuracy, and F1 are all fold-friendly.

The second pattern is to collect confusion matrix terms per fold with cross_validate, which supports callables returning a dictionary of named scores.

The third pattern is to generate out-of-fold predictions with cross_val_predict and then build one confusion matrix from those predictions. That gives you a single matrix for the entire dataset without evaluating on the training data.

Option 1: Use a Derived Scalar Metric

If your real goal is model selection, a scalar metric is usually best:

1from sklearn.datasets import make_classification
2from sklearn.linear_model import LogisticRegression
3from sklearn.metrics import make_scorer, recall_score
4from sklearn.model_selection import StratifiedKFold, cross_val_score
5
6X, y = make_classification(
7    n_samples=400,
8    n_features=10,
9    n_informative=5,
10    weights=[0.8, 0.2],
11    random_state=7,
12)
13
14model = LogisticRegression(max_iter=1000)
15cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
16
17recall_scorer = make_scorer(recall_score)
18scores = cross_val_score(model, X, y, scoring=recall_scorer, cv=cv)
19
20print(scores)
21print(f"Mean recall: {scores.mean():.3f}")

This is the cleanest choice when you need a single number for comparison or hyperparameter search.

Option 2: Return Confusion Matrix Counts Per Fold

If you specifically want confusion matrix information from each fold, use cross_validate with a custom callable:

1import numpy as np
2from sklearn.datasets import make_classification
3from sklearn.linear_model import LogisticRegression
4from sklearn.metrics import confusion_matrix
5from sklearn.model_selection import StratifiedKFold, cross_validate
6
7X, y = make_classification(
8    n_samples=300,
9    n_features=8,
10    n_informative=4,
11    weights=[0.7, 0.3],
12    random_state=42,
13)
14
15def confusion_terms(estimator, X_fold, y_fold):
16    y_pred = estimator.predict(X_fold)
17    tn, fp, fn, tp = confusion_matrix(y_fold, y_pred).ravel()
18    return {
19        "tn": float(tn),
20        "fp": float(fp),
21        "fn": float(fn),
22        "tp": float(tp),
23    }
24
25model = LogisticRegression(max_iter=1000)
26cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
27
28results = cross_validate(model, X, y, cv=cv, scoring=confusion_terms)
29
30print("TP per fold:", results["test_tp"])
31print("FN per fold:", results["test_fn"])
32print("Total TP:", np.sum(results["test_tp"]))
33print("Total FN:", np.sum(results["test_fn"]))

This gives you the raw ingredients. From there you can compute sensitivity, specificity, or any business-specific cost formula.

Option 3: Build One Honest Confusion Matrix With Out-of-Fold Predictions

This is often the most interpretable approach for reports:

1from sklearn.datasets import make_classification
2from sklearn.linear_model import LogisticRegression
3from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix
4from sklearn.model_selection import StratifiedKFold, cross_val_predict
5
6X, y = make_classification(n_samples=250, n_features=6, random_state=0)
7model = LogisticRegression(max_iter=1000)
8cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
9
10y_pred = cross_val_predict(model, X, y, cv=cv)
11cm = confusion_matrix(y, y_pred)
12
13print(cm)
14ConfusionMatrixDisplay(confusion_matrix=cm).plot()

Each prediction is produced by a model that did not see that sample during training, so the matrix is much more honest than fitting once on the full dataset and evaluating on the same data.

Common Pitfalls

Trying to pass confusion_matrix directly as scoring is the most common mistake. The scoring API expects either a named scorer or a callable with the estimator-style signature.

Another mistake is averaging confusion matrices without thinking about class balance. If fold sizes differ, summing counts across folds is usually more meaningful than averaging raw cells.

Multiclass problems also trip people up. In binary classification, ravel() works because the matrix is 2 x 2. In multiclass classification, the matrix is larger, so write code that handles the full array instead of unpacking four values.

Finally, remember the difference between model selection and diagnosis. If you need to rank models, use a scalar metric. If you need to understand failure modes, inspect the confusion matrix after you have a sensible evaluation workflow.

Summary

• A confusion matrix is not a direct drop-in scorer for cross_val_score because it is not a single scalar.
• Use derived scalar metrics such as recall or F1 when you need one number per fold.
• Use cross_validate with a custom callable if you want fold-wise confusion matrix counts.
• Use cross_val_predict when you want one aggregate confusion matrix built from out-of-fold predictions.
• Treat binary and multiclass cases differently, especially if you unpack matrix cells.