GGUF

GGUF is a file format that packs a language model's weights, tokenizer, and configuration into a single file. It was built for the llama.cpp inference engine and the tools around it. GGUF replaced the older GGML format in August 2023 and is now the standard container for local model weights.

A model at 16-bit precision needs about 2 bytes of memory per parameter, so a 7 billion parameter model runs close to 14 GB. Converted to a 4-bit quantization, commonly labeled Q4_K_M, the same model shrinks to roughly 4 to 4.5 GB with a small quality loss. Because the tokenizer and metadata travel inside the file, there are no separate config files to match up, and one file is the whole model.

The quantization levels are what let a model too large for a card's VRAM run on it anyway, at some cost to quality. The format also supports splitting layers between GPU and system RAM, so a card with less memory than the model needs can still run it, just slower.

Sources

Last verified August 29, 2026.