Introduction

A CNN that reports good accuracy can still fail in ways that matter to users. The practical way to improve a classifier is to inspect the samples it gets wrong, measure where the errors cluster, and then feed that information back into training. In Keras, that workflow is mostly array comparison plus a few plots, but the details matter if you want reproducible analysis instead of ad hoc debugging.

Start with Deterministic Predictions

Before you look for mistakes, make sure your test data and predicted labels are aligned. That sounds obvious, but shuffled generators and reordered file paths are a common source of fake error analysis.

If your test set is already loaded as arrays, the basic prediction flow is simple:

1import numpy as np
2from tensorflow import keras
3
4model = keras.models.load_model("cnn_model.keras")
5
6# Example shapes:
7# x_test: (N, H, W, C)
8# y_test: (N,)
9probs = model.predict(x_test, verbose=0)
10y_pred = probs.argmax(axis=1)

For binary classification with a sigmoid output, do not use argmax. Apply an explicit threshold:

probs = model.predict(x_test, verbose=0).ravel()
y_pred = (probs >= 0.5).astype(np.int64)

Keep the original sample identifiers if you have them. If your test images came from files, store file_paths alongside x_test and y_test so every wrong prediction can be traced back to the original asset.

Collect the Wrong Cases with Useful Metadata

The most useful output is not just an index list. You usually want true label, predicted label, model confidence, and optional path or external id.

1wrong_idx = np.where(y_pred != y_test)[0]
2
3print(f"wrong predictions: {len(wrong_idx)}")
4
5wrong_true = y_test[wrong_idx]
6wrong_pred = y_pred[wrong_idx]
7wrong_conf = probs[wrong_idx, wrong_pred] if probs.ndim == 2 else probs[wrong_idx]

When the model is wrong and very confident, that sample deserves attention. High-confidence failures often indicate one of three issues:

• the label is wrong
• preprocessing differs between train and test
• the model learned a shortcut feature instead of the real concept

It is also worth sorting the wrong cases by confidence:

1order = np.argsort(-wrong_conf)
2top_wrong = wrong_idx[order[:10]]
3
4for idx in top_wrong:
5    print(idx, int(y_test[idx]), int(y_pred[idx]))

This quickly surfaces the failures that most damage trust in the model.

Visualize the Misclassified Samples

A confusion matrix tells you which classes are confused, but raw images tell you why. Build a small grid first and inspect it visually.

1import matplotlib.pyplot as plt
2
3class_names = ["cat", "dog", "horse", "bird"]
4
5rows, cols = 3, 4
6plt.figure(figsize=(12, 9))
7
8for plot_i, sample_idx in enumerate(wrong_idx[: rows * cols], start=1):
9    plt.subplot(rows, cols, plot_i)
10    plt.imshow(x_test[sample_idx].astype("uint8"))
11    plt.title(
12        f"T={class_names[y_test[sample_idx]]}\nP={class_names[y_pred[sample_idx]]}"
13    )
14    plt.axis("off")
15
16plt.tight_layout()
17plt.show()

After that, inspect special slices rather than a random wall of images:

• all mistakes for one confused class pair
• only high-confidence mistakes
• only low-light or low-resolution samples
• only mislabeled files suspected by humans

This is where model improvement usually starts. You may find annotation errors faster than you find architecture errors.

Add Class-Level Error Analysis

Per-sample review is useful, but you also need a class-level summary. A confusion matrix and classification report give you that.

1from sklearn.metrics import classification_report, confusion_matrix
2
3cm = confusion_matrix(y_test, y_pred)
4print(cm)
5
6report = classification_report(y_test, y_pred, target_names=class_names)
7print(report)

If one class is consistently predicted as another, that is usually more actionable than a global accuracy drop. For example, if cats are often predicted as dogs, you might need more training samples for small cats, better cropping, or augmentation that reduces background bias.

Exporting the wrong cases to a table also helps when someone else needs to review them:

1import pandas as pd
2
3records = []
4for idx in wrong_idx:
5    records.append(
6        {
7            "index": int(idx),
8            "true_label": class_names[y_test[idx]],
9            "pred_label": class_names[y_pred[idx]],
10            "confidence": float(wrong_conf[np.where(wrong_idx == idx)[0][0]]),
11        }
12    )
13
14df = pd.DataFrame(records)
15df.to_csv("wrong_predictions.csv", index=False)

In a real project, include file path, dataset split version, and model checkpoint name. Without that context, a CSV of wrong predictions becomes much less useful a week later.

Turn Wrong Cases into a Training Improvement Loop

Error analysis only matters if it changes the next training run. After reviewing wrong cases, write down concrete decisions:

• relabel suspicious samples
• add targeted augmentation for confused classes
• rebalance the training set
• normalize preprocessing between training and inference
• collect more examples of edge cases

For harder cases, add model explanations such as Grad-CAM to the wrong predictions. That can show whether the CNN is focusing on the object or on irrelevant background texture. Even a small batch of reviewed explanations can reveal a shortcut-learning problem quickly.

The key is consistency. Run the same wrong-case analysis after every retrain so you can tell whether the new model fixed the actual failures or merely changed the aggregate score.

Common Pitfalls

• Evaluating on shuffled generator output without preserving sample order, which makes true labels and predictions stop matching.
• Using argmax for a sigmoid binary classifier and getting misleading predicted labels.
• Looking only at total accuracy instead of class confusion and high-confidence errors.
• Reviewing images visually but not exporting reproducible metadata such as sample id and confidence.
• Treating every wrong prediction as model weakness when some cases are actually label or data-quality issues.

Summary

• Wrong-case analysis is one of the fastest ways to improve a CNN after initial training.
• In Keras, the core steps are prediction, label comparison, and indexed inspection of failures.
• High-confidence mistakes are usually more informative than low-confidence borderline cases.
• Use both visual review and class-level metrics such as confusion matrices.
• Save wrong-case metadata so the same failures can drive relabeling, retraining, and regression checks.