Introduction

MNIST is simple enough that a basic model can reach good accuracy quickly, but pushing performance higher still depends on careful preprocessing, architecture choices, and error analysis. If a model stalls below expectations, the best improvements usually come from better regularization and cleaner training procedure rather than from making the network arbitrarily larger.

Start With a Strong Baseline

Before tuning anything, make sure the data pipeline is correct. MNIST images should be normalized to a consistent range and reshaped so the model sees a channel dimension.

1import tensorflow as tf
2from tensorflow import keras
3
4(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
5
6x_train = x_train.astype("float32") / 255.0
7x_test = x_test.astype("float32") / 255.0
8
9x_train = x_train[..., None]
10x_test = x_test[..., None]
11
12print(x_train.shape, x_test.shape)

If this step is wrong, every later experiment becomes noisy. A surprisingly common issue is training on 28 x 28 arrays while the model expects 28 x 28 x 1, which silently pushes people into unnecessary debugging.

Use a Better CNN Instead of a Tiny Dense Network

A small dense network can solve MNIST, but convolutional layers usually perform better because they capture local stroke patterns. A compact CNN with batch normalization and dropout is often enough to move a mediocre model into the high-accuracy range.

1import tensorflow as tf
2from tensorflow import keras
3
4data_augmentation = keras.Sequential(
5    [
6        keras.layers.RandomTranslation(0.08, 0.08),
7        keras.layers.RandomRotation(0.05),
8    ]
9)
10
11model = keras.Sequential(
12    [
13        keras.layers.Input(shape=(28, 28, 1)),
14        data_augmentation,
15        keras.layers.Conv2D(32, 3, padding="same", activation="relu"),
16        keras.layers.BatchNormalization(),
17        keras.layers.Conv2D(32, 3, padding="same", activation="relu"),
18        keras.layers.MaxPooling2D(),
19        keras.layers.Dropout(0.25),
20        keras.layers.Conv2D(64, 3, padding="same", activation="relu"),
21        keras.layers.BatchNormalization(),
22        keras.layers.Conv2D(64, 3, padding="same", activation="relu"),
23        keras.layers.MaxPooling2D(),
24        keras.layers.Dropout(0.25),
25        keras.layers.Flatten(),
26        keras.layers.Dense(128, activation="relu"),
27        keras.layers.Dropout(0.4),
28        keras.layers.Dense(10, activation="softmax"),
29    ]
30)
31
32model.compile(
33    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
34    loss="sparse_categorical_crossentropy",
35    metrics=["accuracy"],
36)

This model is still small enough to train quickly, but it has enough capacity to learn robust stroke features instead of memorizing pixel positions.

Train With Callbacks and Sensible Augmentation

MNIST digits are centered and clean, so augmentation should be mild. Small rotations and translations help the model generalize to imperfect handwriting, but strong distortion can create unrealistic digits and reduce accuracy.

Use callbacks to stabilize training:

1callbacks = [
2    keras.callbacks.EarlyStopping(
3        monitor="val_accuracy",
4        patience=4,
5        restore_best_weights=True,
6    ),
7    keras.callbacks.ReduceLROnPlateau(
8        monitor="val_loss",
9        factor=0.5,
10        patience=2,
11    ),
12]
13
14history = model.fit(
15    x_train,
16    y_train,
17    validation_split=0.1,
18    epochs=20,
19    batch_size=128,
20    callbacks=callbacks,
21    verbose=2,
22)
23
24test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0)
25print(f"Test accuracy: {test_accuracy:.4f}")

Early stopping prevents overtraining once validation accuracy plateaus. Learning-rate reduction helps the optimizer make smaller, more useful updates late in training.

Improve the Model by Studying Mistakes

Once the model is strong, the remaining errors often come from ambiguous digits such as 4 and 9 or sloppy handwriting that looks unusual compared with the training set. That means confusion analysis is more useful than blindly stacking extra layers.

A simple review loop is:

• generate predictions on the test set
• locate misclassified images
• group them by true label and predicted label
• inspect whether the problem is noise, class ambiguity, or underfitting

If many 5 digits are predicted as 3, more data augmentation may help. If validation accuracy is much lower than training accuracy, regularization or a smaller model may be the better fix. If both are low, the model or preprocessing pipeline is probably too weak.

Common Pitfalls

• Over-augmenting MNIST. Large rotations or shifts can turn a valid digit into something that no longer belongs to the original label.
• Ignoring normalization and input shape. Bad preprocessing can cost more accuracy than any optimizer change can recover.
• Evaluating only training accuracy. A model that looks excellent during training may still generalize poorly.
• Making the network deeper without checking whether errors are systematic. Bigger models do not solve mislabeled data or weak preprocessing.
• Applying augmentation to validation or test data. That makes results noisy and harder to compare across runs.

Summary

• Start with correct normalization and the expected input shape before tuning architecture.
• A modest CNN usually outperforms a simple dense network on MNIST.
• Mild augmentation, batch normalization, dropout, and callbacks often improve accuracy more than brute-force model growth.
• Use validation curves and misclassification analysis to decide what to change next.
• Focus on generalization, not only training accuracy, when judging whether the model has improved.