Tracing Randomness in PyTorch Training: Six Entry Points and Control Strategies

Why Can't You Reproduce Results Even After Setting the Seed?

In deep learning experiments, setting a random seed has become almost standard practice. But many developers have encountered this confusion: even after fixing torch.manual_seed(0), two training runs still produce subtle differences. A technical discussion thread in the Reddit community systematically breaks down the six entry points of randomness in the PyTorch training pipeline and provides corresponding control measures. The essence of this problem is that randomness does not come from a single source but is distributed across multiple stages: the Python layer, the data pipeline, model initialization, and GPU operators.

The Six Entry Points of Randomness, One by One

1. Python and NumPy Random State

Data loading, data augmentation, and user-defined code often mix random and numpy.random. Setting only the PyTorch seed is not enough; all three must be fixed simultaneously:

import random, numpy as np, torch
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)

This step is foundational, but far from sufficient.

2. The Ordering Trap of Weight Initialization

Model parameter initialization is controlled by the torch seed, but only if the seed is set before model creation. If the model is built first and the seed is set afterward, initialization has already consumed part of the random number sequence, and subsequent seed setting cannot retroactively correct it. This ordering detail is easily overlooked but is a common cause of reproducibility failures.

3. DataLoader's shuffle Generator

DataLoader with shuffle=True uses a random generator internally. A more robust approach is to explicitly pass in a custom generator:

g = torch.Generator()
g.manual_seed(0)
loader = DataLoader(ds, shuffle=True, generator=g)

This way, the randomness source for data order is fully controllable and will not interfere with other random calls.

torch.Generator is an independent object in PyTorch that manages the state of a random number generator. It encapsulates the underlying pseudo-random number algorithm and its internal state (the seed and the sequence of random numbers already consumed). Each Generator instance has its own state and does not interfere with the global default generator. The advantage of explicitly passing the generator argument in DataLoader is that the shuffle operation only consumes the random sequence of that generator. Therefore, even if you call other random functions in the training loop (such as dropout or data augmentation), the random order of shuffle will not change. This is especially important when strict reproducibility of data loading order is required. In addition, using an independent Generator makes it easier to derive different but reproducible seeds for different processes in distributed training, avoiding all processes loading exactly the same data order.

4. Independent Random States of Multi-Process Workers

When num_workers > 0, each worker has an independent random state, and augmentation libraries called inside workers may seed differently. The standard solution is to use worker_init_fn to derive and set a seed for each worker from a base seed. This step is particularly critical when complex preprocessing such as image augmentation and text perturbation is involved.

5. Order Dependence of Dropout and Random Layers

Random layers such as Dropout use the torch generator and are theoretically covered by the seed. But the premise is that the order of operator calls is exactly the same in both runs. Changing the batch order or the number of calls will change how the random number sequence is consumed, causing results to shift. In other words, fixing the seed only guarantees the same starting point of the sequence, not the same consumption path.

6. GPU Non-Deterministic Operators

Some CUDA operations (for example, atomic-add-based operators) can produce slightly different results on each run even with all seeds fixed. Optional mitigation measures include:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

use_deterministic_algorithms throws an error when it encounters an operator without a deterministic version, which actually becomes a clue for locating the problem. Some CUDA versions also require additional environment variables to enable deterministic matrix multiplication; refer to PyTorch's reproducibility documentation for details. The trade-off is that deterministic mode usually comes with a performance penalty.

CUDA atomic operations are a mechanism that allows multiple threads to safely update the same memory address simultaneously, commonly used in scenarios such as reductions and embedding layer gradient accumulation. Because thread execution order is non-deterministic, the order of floating-point additions may differ between runs, and floating-point addition is not associative, resulting in tiny numerical differences. torch.use_deterministic_algorithms(True) forces the use of deterministic versions of operators (if they exist) and throws an error when encountering operators without a deterministic implementation, thereby exposing hidden sources of non-determinism. Note that this setting needs to be combined with the CUBLAS_WORKSPACE_CONFIG environment variable (e.g., :4096:8) to enable deterministic matrix multiplication; otherwise, some CUDA versions will still throw errors. After enabling deterministic mode, performance typically drops by 5%-20%, depending on the model and operator combination.

Cross-Hardware and Distributed: The Boundaries of Exact Reproducibility

Even after completing all the above settings, results may still vary due to differences in hardware, library versions, or distributed configurations. Treating bitwise reproducibility as a goal that requires continuous testing rather than a default assumption is a more pragmatic attitude. The author's advice is: when comparing two methods, do not rely on a single seed; instead, run multiple seeds and report the mean and dispersion. If the difference between methods is smaller than the fluctuation between seeds, it should not be interpreted as an improvement.

Experiment Logging Habits Worth Cultivating

Record the seed, library versions, commit hash, and data split identifier for each run. This information is extremely valuable when troubleshooting reproducibility issues. The question posed at the end of the discussion thread is also worth pondering: are you pursuing statistical reproducibility across seeds, or strict bitwise reproducibility? The former focuses on the robustness of conclusions, while the latter serves debugging and auditing scenarios; the two correspond to different engineering investments.

Frequently Asked Questions

Q: After setting the seed, results still differ. Where should I check first? A: Prioritize checking the seeding logic of multi-process workers and GPU non-deterministic operators; these two are most easily missed by seed coverage.

Q: Does deterministic mode significantly slow down training? A: There is a performance overhead, and the magnitude depends on the model and operator combination. It is recommended to enable it during debugging and disable it as needed for formal training.

Q: Can bitwise reproducibility still be achieved in distributed training? A: Exact reproducibility across hardware and distributed configurations is very difficult. A more realistic approach is to pursue reproducibility at the statistical level.

One-line Summary

Randomness in PyTorch training is distributed across six stages; setting the torch seed alone is insufficient for reproducibility, and systematic control is needed while distinguishing between statistical reproducibility and bitwise reproducibility.

Paragraph Summary

The article systematically breaks down the six entry points of randomness in the PyTorch training pipeline: Python/NumPy random state, weight initialization ordering, DataLoader's shuffle generator, independent random states of multi-process workers, order dependence of random layers such as Dropout, and GPU non-deterministic operators. Setting only torch.manual_seed is far from enough; you need to also fix random and numpy.random, ensure the seed is set before model creation, explicitly pass a custom Generator to DataLoader, use worker_init_fn to seed each worker, and consider enabling deterministic algorithms. The article emphasizes that bitwise reproducibility across hardware and distributed configurations is difficult to guarantee, and recommends running multiple seeds and reporting mean and dispersion when comparing methods, as well as cultivating the habit of recording seeds, library versions, commit hashes, and data split identifiers.