BF16

BF16, or bfloat16 (brain floating point), is a 16-bit floating point format with 1 sign bit, 8 exponent bits and 7 mantissa bits. Google Brain, an artificial intelligence research group at Google, conceived it for machine learning. It keeps the 8-bit exponent of FP32, so its dynamic range matches FP32, and it gives up mantissa precision instead. Google's Cloud TPU v2 and v3 tensor processing units multiply in BF16 and accumulate in FP32.

FP16 makes the opposite trade, with fewer exponent bits and a 10-bit mantissa. FP16 training typically needs loss scaling, while BF16 comes close to a drop-in replacement for FP32. NVIDIA added BF16 to its tensor cores with the Ampere A100, at the same rate as FP16. An A100 delivers 312 TFLOPS of dense BF16, against 19.5 TFLOPS of FP32 on its CUDA cores. An HGX B300 baseboard with eight Blackwell Ultra GPUs is rated 36 PFLOPS of BF16 with sparsity, or 18 PFLOPS dense.

On Ampere and newer GPUs, train and fine-tune in BF16. On Volta and Turing GPUs, whose tensor cores lack BF16, use FP16 with loss scaling.

Sources

Last verified October 7, 2026.