Prefill vs decode

Prefill is the first phase of large language model (LLM) inference, where the model processes the whole prompt in parallel and builds the key-value (KV) cache. Decode is the second phase, generating the reply one token per step. Prefill is compute-bound, a matrix-matrix operation that saturates the GPU. Decode is memory-bound, a matrix-vector operation limited by how fast weights, keys and values move from memory. Prefill sets the time to first token (TTFT), and decode sets the time per output token (TPOT).

For Llama 2 7B in 16-bit precision, NVIDIA puts the weights near 14 GB and the KV cache for one 4,096 token sequence near 2 GB. Every decode step needs those weights plus all prior keys and values. Disaggregated serving runs the two phases on separate GPUs. NVIDIA Dynamo copies the KV cache straight from prefill GPU memory to decode GPU memory over NVLink or InfiniBand. The 2024 DistServe paper reports 7.4x more requests served, or 12.6x tighter latency targets met, than earlier systems. It kept over 90 percent of requests within those limits.

Split the phases onto separate GPU pools only when sustained concurrent load makes long prompts stall active generation. At low concurrency, one aggregated worker often avoids the KV cache transfer overhead.

Sources

Last verified October 7, 2026.