A common and frustrating experience when working with Keras is saving a trained model, loading it back, and finding that the evaluation accuracy no longer matches what you saw during training. This discrepancy does not mean the model weights were corrupted. It is almost always caused by subtle differences in the evaluation environment. Understanding the root causes will help you deploy models with confidence.

Non-Deterministic Operations

Keras and TensorFlow use operations that can produce slightly different results between runs, especially on GPUs. Parallel floating-point reductions (such as summing across threads) can accumulate rounding errors in different orders depending on thread scheduling.

To reduce this variability, set random seeds at the start of your script:

1import os
2import random
3import numpy as np
4import tensorflow as tf
5
6os.environ["TF_DETERMINISTIC_OPS"] = "1"
7random.seed(42)
8np.random.seed(42)
9tf.random.set_seed(42)

The environment variable TF_DETERMINISTIC_OPS forces TensorFlow to use deterministic implementations of GPU kernels, which eliminates one source of variation at the cost of some speed.

Missing custom_objects During Loading

If your model uses custom layers, custom loss functions, or custom metrics, you must pass them in the custom_objects dictionary when loading. Without this, Keras either raises an error or silently substitutes a default implementation that behaves differently.

1from tensorflow.keras.models import load_model
2
3def focal_loss(y_true, y_pred):
4    gamma = 2.0
5    epsilon = 1e-7
6    y_pred = tf.clip_by_value(y_pred, epsilon, 1.0 - epsilon)
7    loss = -y_true * (1 - y_pred) ** gamma * tf.math.log(y_pred)
8    return tf.reduce_mean(loss)
9
10# Correct: pass custom objects explicitly
11model = load_model("my_model.keras", custom_objects={"focal_loss": focal_loss})

If you are using the newer .keras format (Keras 3), custom objects registered with @tf.keras.utils.register_keras_serializable are resolved automatically. However, explicitly passing custom_objects is still the safest approach.

The compile=False Trap

When you call load_model with compile=False, Keras loads only the architecture and weights without restoring the optimizer state, loss function, or metrics. If you then call model.evaluate() without recompiling, Keras raises an error. If you recompile with different settings, the accuracy metric may be computed differently.

1# Loading without compile
2model = load_model("my_model.keras", compile=False)
3
4# You MUST recompile with the exact same settings
5model.compile(
6    optimizer="adam",
7    loss="categorical_crossentropy",
8    metrics=["accuracy"],
9)
10
11loss, acc = model.evaluate(test_ds)

Make sure the loss, metrics, and any class weights match exactly what you used during training. Even switching from "accuracy" to tf.keras.metrics.CategoricalAccuracy() can produce different numeric results due to floating-point accumulation order.

Batch Normalization Inference Mode

Batch normalization layers behave differently during training and inference. During training, they normalize using the current batch statistics. During inference, they use the running mean and variance accumulated over all training batches.

If you accidentally evaluate in training mode, the batch statistics from your test data will be used instead, producing different accuracy numbers. This can happen if you call the model with training=True by mistake:

1# Wrong: forces training mode, BN uses batch statistics
2predictions = model(test_images, training=True)
3
4# Correct: inference mode, BN uses running statistics
5predictions = model(test_images, training=False)
6
7# Also correct: model.predict() uses inference mode by default
8predictions = model.predict(test_images)

Always use model.predict() or explicitly pass training=False when evaluating.

Evaluation Data Shuffle

If your evaluation data is shuffled differently between the original evaluation and the post-load evaluation, accuracy computed on individual batches will differ (though the overall accuracy across the full dataset should converge). More importantly, if you only evaluate on a subset of the data, the shuffle order determines which samples are included.

1# Ensure deterministic evaluation
2test_ds = tf.data.Dataset.from_tensor_slices((test_images, test_labels))
3test_ds = test_ds.batch(32)  # no shuffle for evaluation
4
5loss, acc = model.evaluate(test_ds)

Never shuffle your test dataset. If you loaded it with image_dataset_from_directory, pass shuffle=False explicitly.

Common Pitfalls

• Evaluating on different data subsets. If you use a generator or a shuffled dataset, the samples seen during evaluation may differ between runs. Always disable shuffling for test data.
• Forgetting to set TF_DETERMINISTIC_OPS. Without this environment variable, GPU reductions can produce slightly different results on each run, even with all random seeds fixed.
• Using compile=False and forgetting to recompile. The model loads without metrics, so evaluate() either fails or returns meaningless numbers. Always recompile with the identical configuration.
• Saving with model.save_weights() instead of model.save(). The weights-only format does not save the optimizer state or architecture. You must reconstruct the model in code and call load_weights(), which is error-prone.
• Not pinning library versions. A TensorFlow minor version upgrade can change the default behavior of layers like BatchNormalization or the implementation of certain loss functions. Pin your dependency versions for reproducible results.

Summary

• Non-deterministic GPU operations cause small accuracy variations. Set TF_DETERMINISTIC_OPS=1 and fix all random seeds to minimize them.
• Always pass custom_objects when loading models that use custom layers, losses, or metrics.
• If you use compile=False, recompile with the exact same optimizer, loss, and metrics before evaluating.
• Batch normalization layers must run in inference mode during evaluation. Use model.predict() or pass training=False.
• Disable shuffling on your test dataset so that evaluation results are consistent across runs.