Continuous Batching

Continuous batching allows new requests to be inserted and completed requests to be removed at any iteration step, keeping the GPU running at full load at every step.

Continuous Batching adopts iteration-level scheduling granularity, allowing completed requests to be removed and new requests to be inserted upon each token generation completion.

Inference throughput optimization relies on technologies such as KV Cache management and Continuous Batching, which directly determine response speed and service cost under concurrent user loads.

Continuous Batching allows an inference engine to dynamically merge multiple requests within a single forward pass to maximize GPU utilization.

Increasing batch size to improve throughput typically sacrifices per-request latency

vLLM has batch processing, continuous batching, and memory management capabilities.

In language model inference optimization, common approaches fall roughly into two categories: one reduces the computation and memory overhead of a single inference (such as quantization, knowledge distillation, sparsification, and pruning), and the other reduces redundant computation and improves throughput (such as KV cache reuse, prefix caching, continuous batching, and speculative decoding).

PagedAttention solves the problem of memory-space fragmentation, and continuous batching solves the problem of resource release over time; together they constitute vLLM's throughput advantage.

Common inference optimization methods in the industry include weight quantization (INT8/FP8/INT4), KV Cache reuse and paging, continuous batching, speculative decoding, and others.

Continuous batching dynamically inserts and removes requests at each decoding step, and can significantly improve throughput compared with static batching.

Continuous batching dynamically schedules requests at token granularity, allowing throughput to increase severalfold under real traffic.

With continuous batching, vLLM reaches throughput of 1850+ tok/s at 32 concurrent users, and it is the only one of the three engines designed for multi-tenant API serving.

vLLM's continuous batching mechanism causes the same request to potentially produce different outputs when run at different times, even with temperature=0 and unchanged model weights and code.

vLLM's continuous batching causes the same request, run at different times, to join batches with different compositions, as the lengths, counts, and generation stages of other sequences in the batch are all changing, thereby affecting the computation path.

The trend of integrating inference and training refers to training and inference sharing model definitions, parallel strategies and scheduling infrastructure to reduce duplicated construction of two systems; capabilities accumulated by vLLM on the inference side, such as PagedAttention and continuous batching, are gradually extending into the post-training stage.

Continuous batching dynamically packs requests from different users, causing the same prompt to land on different compute paths at different times.

vLLM dynamically schedules across requests via continuous batching to avoid GPU idling, and pairs this with PagedAttention to compress cache overhead.

vLLM and SGLang are server-oriented inference engines whose core optimizations include PagedAttention and continuous batching, suited to NVIDIA server scenarios.

The scheduling unit of traditional inference services is a single request, with typical strategies such as continuous batching, focusing on maximizing throughput and minimizing P99 latency; whereas in the agent scenario the scheduling unit becomes a task spanning tens of seconds or even minutes and containing multiple model calls and external IO.

Continuous batching dynamically removes completed sequences and inserts new requests after each forward iteration, keeping the GPU fully loaded, and throughput can be increased several times compared with static batching.