Introduction

Setting seeds is necessary for reproducible Keras training, but it only controls one source of randomness. Real training runs can still drift because of nondeterministic GPU kernels, shuffled input pipelines, thread scheduling, and environment differences between machines.

Seed Every Random Source Early

A good reproducibility baseline starts before you build the model or dataset. You want Python, NumPy, and TensorFlow all seeded from the same value.

1import os
2import random
3import numpy as np
4import tensorflow as tf
5
6SEED = 12345
7
8os.environ["PYTHONHASHSEED"] = str(SEED)
9random.seed(SEED)
10np.random.seed(SEED)
11tf.keras.utils.set_random_seed(SEED)
12
13try:
14    tf.config.experimental.enable_op_determinism()
15except Exception:
16    pass

This is the right first step, but it does not guarantee identical results by itself.

Data Pipelines Can Reintroduce Nondeterminism

A tf.data pipeline can still change batch order or execution timing even when the core random generators are seeded. Shuffle behavior is the most obvious example.

1import tensorflow as tf
2
3seed = 12345
4
5dataset = tf.data.Dataset.range(1000)
6dataset = dataset.shuffle(
7    buffer_size=1000,
8    seed=seed,
9    reshuffle_each_iteration=False
10)
11
12options = tf.data.Options()
13options.experimental_deterministic = True
14dataset = dataset.with_options(options)
15dataset = dataset.batch(32)

If the dataset is shuffled differently across runs, training diverges quickly because stochastic optimization is path-dependent.

Parallel mapping can also affect determinism. If strict repeatability matters, keep the input pipeline conservative until you have a verified baseline.

Hardware and Kernels Matter

Even with identical code and seeds, GPU execution can differ from CPU execution, and one GPU model can differ from another. Some low-level operations use nondeterministic kernels for performance reasons, especially in highly parallel reductions.

That means reproducibility is not just about source code. It also depends on:

• TensorFlow version
• Keras version
• CUDA and cuDNN versions
• GPU model and driver
• CPU math libraries
• mixed-precision settings

For teams that need repeatable training, containerizing the environment often matters as much as seed setting.

Training Configuration Still Influences Repeatability

Some Keras defaults add variability if you are not careful. For example, fit(..., shuffle=True) changes sample order unless your input pipeline is already controlled.

1from tensorflow import keras
2
3model = keras.Sequential([
4    keras.layers.Input(shape=(1,)),
5    keras.layers.Dense(16, activation="relu"),
6    keras.layers.Dense(1, activation="sigmoid")
7])
8
9model.compile(optimizer="adam", loss="binary_crossentropy")
10
11history = model.fit(
12    dataset,
13    epochs=3,
14    verbose=0,
15    shuffle=False
16)

The exact right setting depends on how the dataset is built, but the larger lesson is that training configuration can silently add or remove nondeterminism.

Test Reproducibility Instead of Assuming It

A practical way to verify your setup is to train twice in one controlled process and compare weights or metrics.

1import numpy as np
2import tensorflow as tf
3from tensorflow import keras
4
5
6def train_once():
7    tf.keras.backend.clear_session()
8    tf.keras.utils.set_random_seed(12345)
9
10    model = keras.Sequential([
11        keras.layers.Input(shape=(1,)),
12        keras.layers.Dense(8, activation="relu"),
13        keras.layers.Dense(1)
14    ])
15    model.compile(optimizer="sgd", loss="mse")
16
17    x = np.arange(200, dtype=np.float32).reshape(-1, 1)
18    y = (x % 7) / 7.0
19
20    model.fit(x, y, epochs=2, verbose=0, shuffle=False)
21    return model.get_weights()
22
23w1 = train_once()
24w2 = train_once()
25print(all(np.allclose(a, b) for a, b in zip(w1, w2)))

This kind of harness turns reproducibility from a vague hope into a checkable property.

Decide What Level of Reproducibility You Need

Some teams need byte-for-byte identical weights for regression tests. Others only need statistically stable metrics across runs. These are not the same goal.

Strict determinism can require giving up some performance features. If all you need is reliable model quality rather than identical binary output, your controls may be less strict. Define the target before optimizing toward it.

Common Pitfalls

Setting seeds after model or dataset creation is too late for some sources of randomness.

Assuming seeded code is enough while leaving tf.data ordering uncontrolled is a common source of drift.

Comparing runs across different TensorFlow, CUDA, or driver versions can hide environment differences behind what looks like a seed problem.

Summary

• Seed setting is necessary but not sufficient for Keras reproducibility.
• Control the data pipeline, deterministic ops, and environment versions as well as the random generators.
• Build a reproducibility test harness instead of assuming that seeds solved the problem.
• Be explicit about whether you need exact determinism or only stable model quality.