Why does this PyTorch model produce non-identical outputs on identical inputs even with fixed random seeds?

Question

I am running inference on a trained PyTorch model using the same input tensor, fixed random seeds, and evaluation mode enabled.

import torch
torch.manual_seed(42)
torch.cuda.manual_seed_all(42)
model.eval()

Despite this, repeated inference calls produce slightly different outputs at the floating-point level.

Question
Which PyTorch or CUDA operations are non-deterministic during inference, and what exact configuration is required to guarantee deterministic results across runs?

The ordering that threads (warps actually) execute in is indeterminate. — Johan
– Johan, Commented Dec 29, 2025 at 18:22

paleonix · Accepted Answer · 2025-12-30 08:36:02Z

1

Even with fixed random seeds, PyTorch uses non-deterministic cuDNN and CUDA operations by default, so you need to add

torch.use_deterministic_algorithms(True),
torch.backends.cudnn.deterministic = True,
torch.backends.cudnn.benchmark = False,
and set os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'

to ensure completely deterministic inference results.

edited Dec 30, 2025 at 8:36

paleonix

3,3535 gold badges20 silver badges42 bronze badges

answered Dec 30, 2025 at 7:21

user28199534

112 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why does this PyTorch model produce non-identical outputs on identical inputs even with fixed random seeds?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related