Wide Expert Parallelism
A parallelization deployment strategy for Mixture-of-Experts (MoE) models that spreads experts horizontally across a larger number of devices, with each device hosting independent expert weights. By expanding the device pool, this approach achieves load balancing and is particularly effective for large models with high expert counts.
Timeline (last 90 days)
DeepInfer's wide expert parallelism spreads 384 experts across 128 chips, with each chip holding only about 7 experts, reducing memory traffic by nearly 88%, and a single GPU can output more than 26,000 words per second.
At the GPU level, with wide expert parallelism (wide-EP), the expert load carried on different GPUs can vary enormously, causing some GPUs to become bottlenecks and slowing the processing of the entire batch.
Wide expert parallelism (wide-EP) means splitting experts across more GPUs (i.e., a large EP scale) so that each GPU carries fewer experts, which can reduce per-card memory pressure but amplifies the risk of load imbalance.
All Facts (3)
DeepInfer's wide expert parallelism spreads 384 experts across 128 chips, with each chip holding only about 7 experts, reducing memory traffic by nearly 88%, and a single GPU can output more than 26,000 words per second.
50%UnverifiedAt the GPU level, with wide expert parallelism (wide-EP), the expert load carried on different GPUs can vary enormously, causing some GPUs to become bottlenecks and slowing the processing of the entire batch.
50%UnverifiedWide expert parallelism (wide-EP) means splitting experts across more GPUs (i.e., a large EP scale) so that each GPU carries fewer experts, which can reduce per-card memory pressure but amplifies the risk of load imbalance.
50%