KV cache

The KV cache holds the key and value tensors a transformer has already computed for every token it processed. Reusing them saves recomputing attention from scratch at each step. Without the cache, generating token 500 would mean reprocessing all 499 tokens ahead of it. With it, the model computes only the new token and appends the result. The cost is memory, since the cache grows with every token generated and every sequence held in a batch.

Size depends on architecture, context length, and batch size. A 7 billion parameter model with 32 layers, running a 4,096 token context at 16-bit precision, needs roughly 2 GB of cache for one sequence. Serve 32 sequences at once and that reaches 64 GB, well past what the 14 GB of weights consume. Grouped query attention and multi-query attention exist to shrink the cache by sharing key and value heads across queries.

Budget the KV cache separately from the weights when sizing GPU memory for serving. It scales with concurrent users and context length, not with model size, so a long-context deployment can need more memory for cache than for the model itself.

Sources

Last verified August 29, 2026.