INT4

INT4 is a 4-bit signed integer format that stores 16 values, from -8 to 7, in half a byte. Its main use is weight-only quantization. NVIDIA TensorRT, for example, accepts INT4 for weights and not for activations. Each block of 64 or 128 weights shares one 16-bit or 32-bit floating point scale. Multiplying the integer by that scale recovers the approximate weight.

With one 16-bit scale per 128 weights, INT4 costs 4.125 bits per weight. A 70 billion parameter model then needs about 36 GB for weights, against 140 GB in FP16. Native INT4 tensor core math depends on the GPU generation. NVIDIA's Turing architecture added it, and an A100 is rated 1,248 TOPS (trillion operations per second) of dense INT4, double its 624 TOPS of INT8. NVIDIA's tensor core precision list for Blackwell and Rubin includes NVFP4 and INT8 but not INT4. NVFP4 is the 4-bit floating point alternative, with values from -6 to 6 and one scale per block of 16.

When GPU memory, not compute, limits the model, store weights in INT4 to cut weight memory to about a quarter of FP16. On Blackwell or Rubin, test NVFP4 as well, since it is the 4-bit format their tensor cores run natively.

Sources

Last verified October 7, 2026.