Quantization

Quantization reduces the numerical precision of a model's weights, converting them from the 16-bit floating point format most models ship in down to 8-bit or 4-bit integers. It shrinks the file on disk and the memory the model occupies once loaded, since each parameter takes half or a quarter as many bits. The cost is precision. A quantized weight is an approximation of the original, and the model gives up some accuracy for the smaller footprint and faster inference.

A 7 billion parameter model at 16-bit precision needs about 14 GB of memory, 2 bytes for every parameter. Quantized to 8-bit it drops to roughly 7 GB. Quantized to 4-bit, the usual choice for local inference, it lands near 4 GB once scale-factor overhead is counted. GPTQ, AWQ, and the k-quant schemes packaged in GGUF files all do this conversion, and output quality holds up well at 4-bit for everyday work. Degradation gets noticeable below that, especially on math and multi-step reasoning.

Quantization is what makes a large model run on one or two GPUs instead of a rack of them. Check which level a benchmark or vendor claim used before comparing it against a 16-bit number. They are not the same model.

Sources

Last verified August 29, 2026.