Introduction

A confusion matrix needs predicted labels and true labels for the same samples. The important detail is that cross_validate reports scores for each fold, but it does not directly give you one out-of-fold prediction per sample, so it is not enough on its own to build a single cross-validated confusion matrix.

Why cross_validate Alone Is Not Enough

cross_validate is designed to return metrics such as accuracy, precision, or F1 across folds. That is useful for aggregate evaluation, but a confusion matrix requires the predicted class for every sample.

For that reason, the usual solution is:

• use cross_validate if you want fold-level scores
• use cross_val_predict to generate out-of-fold predictions for all rows
• pass those predictions into confusion_matrix

A Practical Pattern

Here is a complete example with scikit-learn:

1from sklearn.datasets import load_iris
2from sklearn.ensemble import RandomForestClassifier
3from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
4from sklearn.model_selection import StratifiedKFold, cross_val_predict, cross_validate
5import matplotlib.pyplot as plt
6
7X, y = load_iris(return_X_y=True)
8
9model = RandomForestClassifier(random_state=42)
10cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
11
12scores = cross_validate(
13    model,
14    X,
15    y,
16    cv=cv,
17    scoring=["accuracy", "precision_macro", "recall_macro"],
18)
19
20predictions = cross_val_predict(model, X, y, cv=cv)
21cm = confusion_matrix(y, predictions)
22
23print("Mean accuracy:", scores["test_accuracy"].mean())
24print(cm)
25
26ConfusionMatrixDisplay(confusion_matrix=cm).plot()
27plt.show()

cross_val_predict trains on each training fold and predicts the held-out fold, repeating until every sample has exactly one prediction from a model that did not train on that sample. That is exactly what you want for a cross-validated confusion matrix.

Per-Fold Confusion Matrices

Sometimes you want one confusion matrix per fold rather than one combined matrix. In that case, use a manual loop over the cross-validation splitter.

1from sklearn.base import clone
2from sklearn.metrics import confusion_matrix
3from sklearn.model_selection import StratifiedKFold
4import numpy as np
5
6cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
7fold_matrices = []
8
9for train_idx, test_idx in cv.split(X, y):
10    estimator = clone(model)
11    estimator.fit(X[train_idx], y[train_idx])
12    preds = estimator.predict(X[test_idx])
13    fold_matrices.append(confusion_matrix(y[test_idx], preds))
14
15mean_matrix = np.mean(fold_matrices, axis=0)
16print(mean_matrix)

This gives you more detail about variability between folds, which can be useful when your dataset is small or imbalanced.

Interpreting the Result

For a combined cross-validated confusion matrix, each row represents the true class and each column represents the predicted class. Because the predictions are out-of-fold, the matrix is a more honest summary than a confusion matrix built on the same data the model trained on.

That distinction matters. A training-set confusion matrix can look excellent even when generalization is poor. Cross-validated predictions are much closer to how the model behaves on unseen data.

Common Pitfalls

The biggest mistake is fitting the model on the full dataset and then calling predict on that same dataset to make the confusion matrix. That evaluates memorization, not generalization.

Another pitfall is assuming cross_validate(return_estimator=True) solves the whole problem. It gives you one trained estimator per fold, but you still need to pair each estimator with its held-out samples if you want fold-specific predictions.

Class imbalance can also distort interpretation. A confusion matrix should usually be read alongside class-wise precision, recall, or normalized counts, especially when one class dominates the dataset.

Finally, make sure you use the same splitter for both metrics and predictions. If cross_validate and cross_val_predict use different folds, the reported scores and the confusion matrix no longer describe the same evaluation process.

Summary

• 'cross_validate gives you fold scores, not the per-sample predictions needed for a confusion matrix.'
• Use cross_val_predict to generate out-of-fold predictions across the dataset.
• Pass those predictions to confusion_matrix for a proper cross-validated matrix.
• Use a manual cross-validation loop when you need one matrix per fold.
• Keep the splitter consistent so your metrics and confusion matrix describe the same experiment.