Mixed precision

Mixed precision is a technique that runs most neural network math in 16-bit formats, FP16 or BF16, while keeping numerically sensitive steps in FP32. In training, an FP32 master copy of the weights accumulates each update, and the forward and backward passes use 16-bit copies. FP16 training also applies loss scaling. It multiplies the loss by a scale factor so small gradients do not flush to zero.

Half precision halves the bytes accessed, which speeds up memory-limited layers. NVIDIA reports up to a 3x overall training speedup on the most arithmetic-intensive models. In its Multibox SSD example, 31% of activation gradient values became zero in FP16, and scaling by 8 recovered FP32 accuracy. FP16 tops out at 65,504. On an A100, FP16 and BF16 tensor cores run up to 16 times faster than FP32 and up to 2 times faster than TF32. In PyTorch, torch.autocast picks a dtype per operation. Linear and convolution layers run in 16-bit, while ops such as sum and softmax stay in float32. The torch.amp.GradScaler class handles FP16 loss scaling.

On Ampere and newer GPUs, autocast to BF16, which also suits BF16-pretrained models that overflow in FP16. On Volta and Turing GPUs, autocast to FP16 and add GradScaler.

Sources

Last verified October 7, 2026.