Introduction

GPU training is often fast precisely because work is parallel, asynchronous, and heavily optimized. Those same properties can make exact reproducibility difficult. Handling non-determinism well means reducing the sources you control, documenting the ones you cannot remove, and deciding whether bit-for-bit repeatability is actually required.

Understand Where Non-Determinism Comes From

Common sources include:

• random initialization and data shuffling
• non-deterministic GPU kernels
• floating-point reduction order
• asynchronous execution
• library and driver differences
• multi-worker or distributed training

Even if you set every random seed, you may still see small training differences because some GPU operations are not deterministic by default.

Set All Relevant Random Seeds

The first step is always seed control.

TensorFlow example:

1import os
2import random
3import numpy as np
4import tensorflow as tf
5
6seed = 1234
7os.environ["PYTHONHASHSEED"] = str(seed)
8random.seed(seed)
9np.random.seed(seed)
10tf.random.set_seed(seed)

PyTorch example:

1import random
2import numpy as np
3import torch
4
5seed = 1234
6random.seed(seed)
7np.random.seed(seed)
8torch.manual_seed(seed)
9torch.cuda.manual_seed_all(seed)

This does not solve everything, but without it reproducibility usually fails immediately.

Enable Deterministic Framework Settings

Most major frameworks expose deterministic modes.

TensorFlow:

import tensorflow as tf

tf.config.experimental.enable_op_determinism()

PyTorch:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

These settings may slow training or disallow certain fast kernels, but they are often the most direct way to reduce run-to-run variation.

Keep the Data Pipeline Reproducible

Data loading is an easy place to lose determinism.

If you shuffle a dataset, make the shuffle seed explicit:

dataset = dataset.shuffle(10000, seed=1234, reshuffle_each_iteration=False)

If you use multiple workers in a data loader, check whether worker initialization and ordering are deterministic. Seeding only the model is not enough if the sample order changes silently.

Expect Tradeoffs Between Speed and Reproducibility

Deterministic execution is rarely free. Some high-performance kernels use algorithms whose operation order is not fixed, especially for reductions and atomic updates. Replacing them with deterministic alternatives can reduce throughput.

That means you should decide what you really need:

• exact repeatability for debugging or regression tests
• approximate reproducibility for research reporting
• maximum throughput for production training

Those goals are related, but they are not identical.

Fix the Software Stack When Comparing Runs

If one experiment uses different versions of CUDA, cuDNN, TensorFlow, PyTorch, or GPU drivers, non-determinism is not the only problem. You are no longer comparing the same execution environment.

For reproducible experiments, record at least:

• framework version
• CUDA version
• cuDNN version
• driver version
• GPU model
• operating system

Containerized environments help because they reduce hidden drift in the software stack.

Be Realistic About Floating-Point Math

Floating-point addition is not associative. That means parallel reductions can produce slightly different results depending on operation order.

For example, mathematically equivalent orderings can differ in floating-point arithmetic:

a = (1e20 + -1e20) + 3.14
b = 1e20 + (-1e20 + 3.14)
print(a, b)

This matters in GPU kernels where parallel execution changes the effective reduction order.

So some variation is not a bug in the usual sense. It is a consequence of how finite-precision math behaves under parallel execution.

Save and Reuse More Than Just Seeds

For experiments you truly need to reproduce, save:

• model weights
• optimizer state
• preprocessing configuration
• dataset split definitions
• random seeds

A seed alone does not recreate the full experiment if other state is missing.

Test for Stability, Not Just Identity

In many practical workflows, you do not need every run to match bit-for-bit. You need the training process to be stable within an acceptable tolerance.

That is why it is often better to track:

• mean and variance across repeated runs
• validation metric bands
• acceptable loss deltas

This produces a more realistic standard than demanding exact matching from a highly parallel floating-point system.

Common Pitfalls

• Setting one seed and assuming the entire training stack is now deterministic.
• Ignoring data-loader ordering and shuffle behavior.
• Forgetting that framework, CUDA, or driver changes can alter results even with the same code.
• Turning on deterministic settings without expecting the possible speed penalty.
• Treating tiny floating-point differences as proof that the training loop is fundamentally broken.

Summary

• GPU training non-determinism comes from randomness, parallel kernels, floating-point math, and software-stack variation.
• Set all relevant seeds, not just the framework seed.
• Enable deterministic framework settings when exact reproducibility matters.
• Keep the data pipeline and environment stable as part of the experiment design.
• Decide whether you need exact bitwise determinism or simply stable, reproducible results within a tolerance.