DWDP
A novel MoE prefill parallelism strategy proposed for SGLang that uses NVLink P2P prefetching to preload peer expert weights and compute all experts locally, eliminating the all-to-all token dispatch overhead inherent in expert parallelism and significantly improving MoE model prefill throughput.
Timeline (last 90 days)
SGLang introduces DWDP (Distributed Weight Data Parallel) prefill parallelism strategy, which prefetches peer expert weights via NVLink P2P and computes locally, eliminating the all-to-all token dispatch overhead of EP.
In a prefill-only test running gpt-oss-120b on 4× B200s, DWDP4 achieves a 1.92× speedup over DEP4 under the MNT 32K/ISL 32K configuration and a 1.54× throughput improvement at saturation (506K vs. 329K tok/s).
All Facts (2)
SGLang introduces DWDP (Distributed Weight Data Parallel) prefill parallelism strategy, which prefetches peer expert weights via NVLink P2P and computes locally, eliminating the all-to-all token dispatch overhead of EP.
50%UnverifiedIn a prefill-only test running gpt-oss-120b on 4× B200s, DWDP4 achieves a 1.92× speedup over DEP4 under the MNT 32K/ISL 32K configuration and a 1.54× throughput improvement at saturation (506K vs. 329K tok/s).
50%