Prefill-Decode Disaggregation
An architectural strategy in LLM inference systems that deploys the prefill stage and the decode stage on separate resources. By independently scaling compute for each stage, this approach optimizes overall inference throughput and latency.
Timeline (last 90 days)
Common optimization strategies for high-throughput prefill include chunked prefill, prefix cache reuse, and PD disaggregation (decoupled deployment of prefill/decode).
DeepSeek adopts a PD-separated architecture, deploying the Pre-fill stage and the Decode stage on different machines: an 8-card server for reading the question and a 64-to-128-card supernode for writing the answer.
Distributed inference parallelism strategies include tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and PD disaggregation.
Chunked prefill alleviates blocking by interleaving prefill slices with decode, but prefill and decode still share the same HBM bandwidth and SMs, so resource contention cannot be fundamentally eliminated.
In a toy simulation with a 1B-class model and 16K to 64K token prompts, PD disaggregation reduced p99 inter-token latency from 5.0ms to 1.0ms and the worst-case stall from 931ms to 1ms.
If a team is already deploying vLLM with PD disaggregation, bidirectional KV transfer is a natural upgrade; if it is still deploying on a single node, it needs to consider the network dependency introduced by cross-node transfer, because high-speed interconnects like EFA are not available in all environments.
PD disaggregation deploys the compute-intensive Prefill stage and the memory-bandwidth-intensive Decode stage of inference on separate instances for independent scaling and optimization
PD separation breaks data locality, as the KV cache generated by prefill nodes needs to be transferred to decode nodes over the network, with bandwidth consumption and latency increasing as input sequence length grows
OpenAI has not adopted a prefill-decode separated PD architecture on its in-house chips
Prefill-Decode (PD) disaggregation requires transferring KV Cache from the Prefill instance to the Decode instance; a CX7 NIC at full 400 Gbps IB delivers approximately 50 GB/s, and transferring 40 GB of KV Cache takes 0.8 seconds.
1 more timeline events
All Facts (11)
Common optimization strategies for high-throughput prefill include chunked prefill, prefix cache reuse, and PD disaggregation (decoupled deployment of prefill/decode).
50%UnverifiedDeepSeek adopts a PD-separated architecture, deploying the Pre-fill stage and the Decode stage on different machines: an 8-card server for reading the question and a 64-to-128-card supernode for writing the answer.
50%UnverifiedDistributed inference parallelism strategies include tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and PD disaggregation.
50%UnverifiedChunked prefill alleviates blocking by interleaving prefill slices with decode, but prefill and decode still share the same HBM bandwidth and SMs, so resource contention cannot be fundamentally eliminated.
50%UnverifiedIn a toy simulation with a 1B-class model and 16K to 64K token prompts, PD disaggregation reduced p99 inter-token latency from 5.0ms to 1.0ms and the worst-case stall from 931ms to 1ms.
50%UnverifiedIf a team is already deploying vLLM with PD disaggregation, bidirectional KV transfer is a natural upgrade; if it is still deploying on a single node, it needs to consider the network dependency introduced by cross-node transfer, because high-speed interconnects like EFA are not available in all environments.
50%UnverifiedPD disaggregation deploys the compute-intensive Prefill stage and the memory-bandwidth-intensive Decode stage of inference on separate instances for independent scaling and optimization
50%UnverifiedPD separation breaks data locality, as the KV cache generated by prefill nodes needs to be transferred to decode nodes over the network, with bandwidth consumption and latency increasing as input sequence length grows
50%UnverifiedOpenAI has not adopted a prefill-decode separated PD architecture on its in-house chips
50%UnverifiedPrefill-Decode (PD) disaggregation requires transferring KV Cache from the Prefill instance to the Decode instance; a CX7 NIC at full 400 Gbps IB delivers approximately 50 GB/s, and transferring 40 GB of KV Cache takes 0.8 seconds.
50%UnverifiedSplitwise, DistServe, and Mooncake are all representative implementations of prefill-decode (PD) disaggregation.
50%