Introduction

GPU training often produces slightly different results across runs, even when the code looks identical. That does not always mean something is wrong. It usually means your training stack includes random initialization, parallel kernels, or library routines that are not deterministic by default.

What You Can and Cannot Guarantee

The first practical rule is to narrow the goal. Current official guidance from both PyTorch and TensorFlow makes the same point: you can reduce nondeterminism on the same hardware, with the same software versions, but you cannot assume identical results across different GPUs, driver stacks, framework releases, or CPU-versus-GPU execution.

That distinction matters because many teams spend hours chasing "nondeterminism" that is really a cross-environment reproducibility problem. Start by deciding whether you need:

• identical reruns on one machine for debugging
• stable training curves for experiments on one platform
• comparable behavior across machines, which is harder and often impossible to guarantee exactly

A Practical PyTorch Recipe

PyTorch documents two main steps: seed all relevant random number generators and force deterministic algorithms where possible.

1import os
2import random
3import numpy as np
4import torch
5
6seed = 42
7os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
8
9random.seed(seed)
10np.random.seed(seed)
11torch.manual_seed(seed)
12torch.cuda.manual_seed_all(seed)
13
14torch.backends.cudnn.benchmark = False
15torch.use_deterministic_algorithms(True)
16
17
18def seed_worker(worker_id):
19    worker_seed = torch.initial_seed() % 2**32
20    np.random.seed(worker_seed)
21    random.seed(worker_seed)
22
23
24generator = torch.Generator()
25generator.manual_seed(seed)

torch.use_deterministic_algorithms(True) tells PyTorch to pick deterministic implementations when available and to raise an error when an operation has no deterministic path. Disabling cuDNN benchmarking prevents the library from changing convolution algorithm choices between runs.

If you use a DataLoader with worker processes, pass worker_init_fn=seed_worker and generator=generator so data loading randomness stays repeatable as well.

The TensorFlow Equivalent

TensorFlow exposes a similar path. The official deterministic setup is to set the random seed early and enable op determinism.

1import tensorflow as tf
2
3tf.keras.utils.set_random_seed(42)
4tf.config.experimental.enable_op_determinism()

set_random_seed() covers Python, NumPy, and TensorFlow seeds in one call. enable_op_determinism() tells TensorFlow to use deterministic GPU kernels where supported. As TensorFlow notes in its documentation, this can reduce performance, and some ops may still raise errors or behave differently if no deterministic implementation exists.

Other Sources of Variation

Even after you seed everything, other moving parts can still change results:

• data shuffling or augmentation running in parallel workers
• mixed precision and reduced numerical precision effects
• nondeterministic kernels that do not have a deterministic replacement
• distributed training across multiple GPUs or machines
• using uninitialized memory or depending on iteration order in unordered containers

This is why reproducibility work usually starts with the smallest possible setup: one machine, one GPU, one fixed dataset order, one framework version, and deterministic flags enabled. Once that is stable, you can add performance features back one by one.

Common Pitfalls

• Setting only one random seed and forgetting NumPy, Python, data-loader workers, or framework-specific generators.
• Expecting bit-for-bit identical results across different hardware, driver versions, or framework releases.
• Leaving cuDNN benchmarking enabled while asking for deterministic behavior.
• Ignoring the runtime errors raised when an operation has no deterministic implementation.
• Turning on deterministic modes in production benchmarks without accounting for the performance cost.

Summary

• GPU nondeterminism usually comes from random seeds, parallel execution, and library kernel choices.
• Reproducibility is easiest to achieve on the same machine with the same software stack.
• In PyTorch, seed all RNGs and enable deterministic algorithms.
• In TensorFlow, set the random seed and enable op determinism early in program startup.
• Determinism is a debugging and validation tool, but it often costs speed.