FP16

FP16, or half precision, is a 16-bit floating point format made up of 1 sign bit, 5 exponent bits, and 10 mantissa bits. It stores a value in 2 bytes, half the footprint of FP32, at the cost of a narrower exponent range and fewer significant digits.

FP16 covers roughly plus or minus 65,504 and holds about 3 to 4 decimal digits of precision, tight enough that gradients can underflow to zero during training. Mixed precision training works around this by running the forward and backward passes in FP16 while keeping a master copy of the weights in FP32. BF16 is FP16's sibling: same 16-bit width, but 8 exponent bits and 7 mantissa bits. It matches FP32's range while giving up more precision, sidestepping the underflow problem without needing an FP32 master copy.

For training, BF16 is now the more common default on data center GPUs from Ampere onward, since it needs no loss scaling. FP16 still shows up in inference and on older hardware without native BF16 support.

Sources

Last verified August 29, 2026.