Mega MoE
An operator design approach that fuses dispatch, GEMM and combine in MoE into a single persistent kernel, overlapping communication and computation through warp specialization.
Timeline (last 90 days)
The Mega MoE fused operator makes communication warps and compute warps share the same batch of SMs, causing contention for registers, shared memory, and issue slots, with the synchronization domain covering the entire SM array, and each replay requires clearing the workspace and performing a cross-rank barrier.
Prime Intellect released Prime Mega MoE, adding backpropagation to DeepSeek's fused Mega MoE kernel, making it trainable.
DeepSeek's open-source Mega MoE kernel previously lacked a backpropagation implementation and could not be used directly for training.
After integrating the trainable Mega MoE kernel into prime-rl, both the forward and backward sides of the RL loop use the same fused implementation, avoiding path-switching overhead between sampling and updating.
For other domestic chip vendors (Cambricon, Hygon, Kunlunxin, Moore Threads, MetaX, Biren, Enflame, and Iluvatar CoreX) to fully adapt DeepInfer, they would roughly need to overturn their existing engines and fill in about 30 TileLang backend operators, FP8×FP4 GEMM fusion, Mega MoE, sparse attention, and TopK components.
Mega MoE fuses the originally separate dispatch, compute, and combine steps of MoE inference into one large fused operator for execution, reducing kernel launch overhead and intermediate-result memory round trips and allowing better overlap of communication and computation.
All Facts (6)
The Mega MoE fused operator makes communication warps and compute warps share the same batch of SMs, causing contention for registers, shared memory, and issue slots, with the synchronization domain covering the entire SM array, and each replay requires clearing the workspace and performing a cross-rank barrier.
50%UnverifiedDeepSeek's open-source Mega MoE kernel previously lacked a backpropagation implementation and could not be used directly for training.
50%UnverifiedAfter integrating the trainable Mega MoE kernel into prime-rl, both the forward and backward sides of the RL loop use the same fused implementation, avoiding path-switching overhead between sampling and updating.
50%UnverifiedFor other domestic chip vendors (Cambricon, Hygon, Kunlunxin, Moore Threads, MetaX, Biren, Enflame, and Iluvatar CoreX) to fully adapt DeepInfer, they would roughly need to overturn their existing engines and fill in about 30 TileLang backend operators, FP8×FP4 GEMM fusion, Mega MoE, sparse attention, and TopK components.
50%UnverifiedMega MoE fuses the originally separate dispatch, compute, and combine steps of MoE inference into one large fused operator for execution, reducing kernel launch overhead and intermediate-result memory round trips and allowing better overlap of communication and computation.
50%UnverifiedPrime Intellect released Prime Mega MoE, adding backpropagation to DeepSeek's fused Mega MoE kernel, making it trainable.
50%