When you run the same TensorFlow training script twice and get different loss curves or accuracy numbers, it can make debugging and benchmarking extremely difficult. Non-repeatable results are a well-known challenge in deep learning, and TensorFlow has multiple sources of randomness that you need to control. This article walks through every layer of the problem and gives you a concrete checklist to follow.

Why Results Vary Between Runs

TensorFlow programs involve randomness at multiple levels: Python's built-in random module, NumPy's random number generator, TensorFlow's own RNG, GPU thread scheduling, and data pipeline ordering. If even one of these sources is not pinned down, your results will differ from run to run.

Setting Random Seeds

The first and most important step is to fix seeds for all three random number generators that TensorFlow code typically uses.

1import os
2import random
3import numpy as np
4import tensorflow as tf
5
6# 1. Python built-in
7random.seed(42)
8
9# 2. NumPy
10np.random.seed(42)
11
12# 3. TensorFlow global seed
13tf.random.set_seed(42)

The TensorFlow global seed controls weight initialization, dropout masks, and any other tf.random operation. Without it, every layer that involves randomness will produce different starting conditions.

You can also set operation-level seeds for individual layers if you need fine-grained control:

initializer = tf.keras.initializers.GlorotUniform(seed=42)
layer = tf.keras.layers.Dense(64, kernel_initializer=initializer)

GPU Non-Determinism

Even with all seeds set, GPU computations can still produce different results. This happens because certain CUDA kernels (such as tf.reduce_sum or convolution backward passes) use parallel reductions where the order of floating-point additions varies between runs. Different addition orders produce different rounding errors.

TensorFlow provides an environment variable to force deterministic GPU kernels:

os.environ["TF_DETERMINISTIC_OPS"] = "1"

Set this at the very top of your script, before importing TensorFlow. With this flag enabled, TensorFlow replaces non-deterministic CUDA kernels with deterministic alternatives. The tradeoff is that some operations become slower (sometimes by 2-6x for specific ops like certain backward passes).

Starting with TensorFlow 2.8, you can also use the Python API:

tf.config.experimental.enable_op_determinism()

This call sets the same flag and additionally raises errors if any operation does not have a deterministic implementation, which helps you identify problem spots.

Data Pipeline Shuffle Seeds

The tf.data pipeline is another source of non-determinism. When you call .shuffle(), the order of elements depends on the internal shuffle buffer state. Always pass a seed:

1train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train))
2train_ds = train_ds.shuffle(buffer_size=10000, seed=42)
3train_ds = train_ds.batch(32)
4train_ds = train_ds.prefetch(tf.data.AUTOTUNE)

If you use image_dataset_from_directory or other high-level loaders, pass the seed parameter:

1train_ds = tf.keras.utils.image_dataset_from_directory(
2    "data/train",
3    seed=42,
4    image_size=(128, 128),
5    batch_size=32,
6)

Additionally, if you use .interleave() or .map() with num_parallel_calls, these can introduce ordering variation. Set deterministic=True:

train_ds = train_ds.map(preprocess_fn, num_parallel_calls=tf.data.AUTOTUNE, deterministic=True)

Multi-Thread and Multi-GPU Considerations

When training on multiple GPUs with tf.distribute.MirroredStrategy, the gradient all-reduce step can introduce non-determinism. The TF_DETERMINISTIC_OPS flag covers most cases, but you should also limit CPU parallelism to remove thread-scheduling variation:

tf.config.threading.set_intra_op_parallelism_threads(1)
tf.config.threading.set_inter_op_parallelism_threads(1)

This makes execution single-threaded and therefore deterministic, but significantly slower. Use this setting only when you need exact reproducibility (such as for debugging or academic benchmarks), not for production training.

Reproducibility Checklist

Use this checklist as a quick reference when setting up a reproducible TensorFlow experiment:

1import os
2os.environ["TF_DETERMINISTIC_OPS"] = "1"
3os.environ["PYTHONHASHSEED"] = "42"
4
5import random
6import numpy as np
7import tensorflow as tf
8
9random.seed(42)
10np.random.seed(42)
11tf.random.set_seed(42)
12
13# Optional: single-threaded execution for exact reproducibility
14tf.config.threading.set_intra_op_parallelism_threads(1)
15tf.config.threading.set_inter_op_parallelism_threads(1)

Place this block at the very top of your script, before any model definitions or data loading.

Common Pitfalls

• Setting seeds after importing TensorFlow. Some internal state is initialized at import time. Set environment variables like TF_DETERMINISTIC_OPS and PYTHONHASHSEED before the import tensorflow statement.
• Forgetting the data pipeline seed. Even with model-level seeds fixed, a shuffled dataset without a seed will feed data in a different order each run, producing different gradient updates.
• Assuming CPU execution is deterministic. While CPU ops are generally more deterministic than GPU ops, multi-threaded CPU execution can still produce slightly different results due to thread scheduling.
• Ignoring library version differences. Moving from TensorFlow 2.10 to 2.12, for example, may change default kernel implementations. Always pin your TensorFlow version and CUDA/cuDNN versions in your requirements file.
• Using TF_DETERMINISTIC_OPS in production training. The deterministic kernels are slower. Use them during debugging and benchmarking, but consider whether the performance cost is acceptable for large-scale production training.

Summary

• Fix seeds for Python's random, NumPy, and tf.random.set_seed() at the top of every script.
• Set TF_DETERMINISTIC_OPS=1 (or call tf.config.experimental.enable_op_determinism()) to force deterministic GPU kernels.
• Always pass a seed to tf.data.Dataset.shuffle() and any other data pipeline operation that involves randomness.
• For exact reproducibility, set intra-op and inter-op parallelism threads to 1, accepting the performance tradeoff.
• Pin your TensorFlow and CUDA versions, and place all seed-setting code before the import tensorflow line.