Time to first token (TTFT)

Time to first token, or TTFT, is the delay between sending a request to a language model and receiving the first token of its reply. It covers queueing, the prefill pass over the prompt, and network latency. Prefill must process the whole input to build the key-value (KV) cache before generation starts, so a longer prompt means a longer TTFT. Time per output token (TPOT) is the average gap between the tokens that follow. NVIDIA treats inter-token latency (ITL) as the same metric, while vLLM computes the two separately.

MLPerf Inference caps TTFT and TPOT for its large language model (LLM) tests. Llama 2 70B must meet a 2,000 ms TTFT and a 200 ms TPOT in the conversational category, and 450 ms and 40 ms in the interactive one. DeepSeek-R1 gets 2,000 ms and 80 ms in the server scenario. The vLLM benchmark tool prints mean, median and 99th percentile values for each metric. Concurrency above the maximum batch size times the replica count queues requests and raises TTFT.

For chat or coding assistants, size the GPU against a TTFT target at the longest prompt expected, not against tokens per second alone. Batch jobs that read output only after completion can ignore TTFT and tune for throughput.

Sources

Last verified October 7, 2026.