Transformer Engine

Transformer Engine (TE) is NVIDIA's open-source library for running Transformer models in 8-bit and 4-bit floating point. It supplies PyTorch and JAX modules and a C++ API, and the modules keep the scaling factors that low precision needs. NVIDIA's documentation says FP8 was introduced with the Hopper architecture. TE supports FP8 on Hopper, Ada and Blackwell GPUs, and adds MXFP8 and NVFP4 on Blackwell. The GitHub repository carries an Apache-2.0 license.

FP8 current scaling keeps one FP32 scale factor per tensor, set from the tensor's absolute maximum, and uses E4M3 for the forward pass and E5M2 for the backward pass by default. It needs an Ada or later GPU, SM 8.9 and up. NVFP4 stores 4-bit E2M1 values, with magnitude up to 6, and applies an FP8 E4M3 scale to each block of 16 values plus an FP32 scale for the whole tensor. These recipes are from TE 2.20.2.

Ask which TE recipe a training or serving stack uses, such as FP8 current scaling or NVFP4, rather than whether it uses FP8. Ampere GPUs get FP16 and BF16 optimizations from TE but not FP8. Hopper, Ada and Blackwell are the GPUs where FP8 applies. TE 2.20.2 lists NVFP4 training for SM 10.0 and 10.3 only.

Sources

Last verified October 30, 2026.