Speculative decoding

Speculative decoding is a large language model (LLM) speedup method in which a cheap draft proposes several tokens and the target model checks them in parallel. The target model accepts or rejects each drafted token. In the original method, that acceptance rule keeps the output distribution identical to running the target model alone. The draft can be a small separate model or extra prediction heads on the target, as in Medusa.

Decode normally runs the large model once per token. Its speed depends on how fast weights load from memory, which leaves Tensor Cores underused. Speculative decoding spends that idle compute on verification. The original paper used a 77 million parameter T5-small to draft for an 11 billion parameter T5-XXL. It measured up to 3.4x faster English to German translation at batch size 1. With Medusa heads, NVIDIA measured Llama 3.1 405B at 108 tokens per second per user on an eight-GPU HGX H200. That is over 1.9x the rate without them, at batch size 1.

Enable it for latency-sensitive serving at low batch sizes, where decode is memory-bound and compute sits spare. At high request rates that spare compute shrinks, and vLLM rates a separate draft model's gain as medium rather than high.

Sources

Last verified October 7, 2026.