Tokens per second
Tokens per second measures how fast a model generates output during inference. It's the standard throughput metric for LLM serving.
Tokens per second, or tok/s, measures how many tokens a model produces or processes each second. Generation splits into two phases that run at different speeds. Prefill reads the prompt in parallel, so it is fast. Decode emits one token at a time, and its speed depends on memory bandwidth more than raw compute. Decode is the number people usually quote.
A single high-end GPU running a 7 billion parameter model at 16-bit precision decodes roughly 40 to 70 tokens per second for one user. A 70 billion parameter model needs several GPUs or a 4-bit quantized version to fit at all, and even then runs 10 to 20 tokens per second per user. Batching many requests raises total throughput across users while lowering what any single user sees. Comfortable reading sits near 5 to 7 tokens per second, since a token averages about three-quarters of an English word.
Ask for decode speed at the batch size and context length you will actually run, not a best-case single-token figure. A GPU that wins at batch size 1 can lose at batch size 16, because one workload leans on bandwidth and the other on compute.
Sources
Source | Publisher |
|---|---|
NVIDIA | |
NVIDIA (TensorRT-LLM) | |
vLLM |
- Publisher
NVIDIA
- Publisher
NVIDIA (TensorRT-LLM)
- Publisher
vLLM
Last verified August 29, 2026.
- tok/s
- TPS
- inference throughput
- decode speed
- prefill speed
- generation speed
- LLM benchmarking
- tokens/sec
- latency vs throughput