Continuous batching
Continuous batching is an LLM serving method that adds and drops requests at every decode step instead of waiting for the whole batch to finish.
Continuous batching is a large language model (LLM) serving method that adds requests to the running batch and drops finished ones at every decode step. Static batching holds the whole batch until its longest request finishes, so early finishers cannot return and new arrivals must wait. Orca calls the idea iteration-level scheduling, TensorRT-LLM calls it in-flight batching, and vLLM calls it continuous batching.
The scheduler runs the model for one iteration, producing one token per request, then returns completed requests and admits waiting ones. Orca, the 2022 research system that introduced iteration-level scheduling, delivered 36.9x the throughput of NVIDIA FasterTransformer at the same latency on GPT-3 175B. NVIDIA reports that in-flight batching plus kernel optimizations at least doubled TensorRT-LLM throughput on H100 GPUs in a benchmark of real-world requests. vLLM pairs continuous batching with chunked prefill and prefix caching.
Choose a serving engine with continuous batching for any deployment where several users send requests of different lengths. A single user sending one request at a time gains little, since there is no second request to slot in.
Sources
Source | Publisher |
|---|---|
Orca: A Distributed Serving System for Transformer-Based Generative Models | USENIX |
NVIDIA TensorRT-LLM Supercharges Large Language Model Inference on NVIDIA H100 GPUs | NVIDIA |
vLLM |
- Publisher
vLLM
Last verified October 7, 2026.
- in-flight batching
- iteration-level scheduling
- dynamic batching
- static batching vs continuous batching
- continuous batching meaning
- LLM batching
- selective batching
- Orca OSDI 2022
- vLLM continuous batching
- TensorRT-LLM in-flight batching