TF32

TF32, or TensorFloat-32, is an NVIDIA tensor core math mode that multiplies FP32 data using the 8-bit exponent of FP32 and the 10-bit mantissa of FP16. With the sign bit, the format uses 19 bits. Inputs and outputs stay standard FP32. The tensor core rounds each input to a 10-bit mantissa, multiplies, and accumulates the result in FP32.

NVIDIA introduced TF32 with the Ampere architecture. An A100 reaches 156 TFLOPS of dense TF32, eight times the 19.5 TFLOPS of its FP32 CUDA cores. An H100 SXM reaches 989 TFLOPS of TF32 with sparsity, or about 495 TFLOPS dense, against 67 TFLOPS of FP32. Precision is the cost. In PyTorch's A100 benchmark, a TF32 matrix multiply ran about 7 times faster than FP32. Its error against an FP64 reference was about two orders of magnitude larger. PyTorch has left TF32 off by default for matrix multiplies since version 1.12. Its cuDNN flag, which covers convolutions, still defaults to on.

Turn TF32 on to speed up unmodified FP32 code on Ampere or newer GPUs when a 10-bit mantissa is acceptable. Once the code can change, move to BF16 or FP16 mixed precision, which runs up to 2 times faster than TF32 on an A100.

Sources

Last verified October 7, 2026.