DeepSeek-V2
DeepSeek发布的MoE大语言模型，引入细粒度专家切分与共享专家机制，将专家数量从8扩大至160，部分专家对所有token始终激活以保留通用知识
Core Facts
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
DeepSeek V4 Pro adopts a MoE (Mixture of Experts) architecture, activating only a subset of parameters during inference to reduce computational costs.
DeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
Timeline (last 90 days)
Open-source models such as Mixtral 8×7B and DeepSeek-V2 both adopt MoE architecture.
After 2024, models such as DeepSeek-V2 and Qwen2 began to match or even surpass contemporaneous Western competitors on certain benchmarks.
DeepSeek-V2 has 160 experts.
Recently popular models such as DeepSeek-V2/V3 and Qwen2-MoE all adopt the MoE architecture.
DeepSeek-V2 adopted a configuration of 1 shared expert plus multiple routed experts, achieving results superior to pure routed MoE.
The activation rate of Mixtral 8x7B is approximately 25%, and the activation rate of DeepSeek-V2 is approximately 10%.
DeepSeek-V2将每个Transformer层的FFN拆分为多个专家模块，通过门控网络动态路由，每个token只激活约10%的参数
Mixtral and DeepSeek-V2 adopt sparse activation strategies
All Facts (14)
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedDeepSeek V4 Pro adopts a MoE (Mixture of Experts) architecture, activating only a subset of parameters during inference to reduce computational costs.
80%VerifiedDeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
80%VerifiedDeepSeek-V3 was released at the end of 2024.
75%VerifiedOpen-source models such as Mixtral 8×7B and DeepSeek-V2 both adopt MoE architecture.
65%VerifiedDeepSeek partially addressed MoE engineering challenges in V3 through an auxiliary-loss-free load balancing strategy and a dedicated MoE parallelism framework.
65%UnverifiedMixtral and DeepSeek-V2 adopt sparse activation strategies
90%UnverifiedTypical representatives of the MoE architecture include Mixtral 8x7B and DeepSeek-V2.
60%UnverifiedAfter 2024, models such as DeepSeek-V2 and Qwen2 began to match or even surpass contemporaneous Western competitors on certain benchmarks.
50%UnverifiedDeepSeek-V2 has 160 experts.
50%UnverifiedRecently popular models such as DeepSeek-V2/V3 and Qwen2-MoE all adopt the MoE architecture.
50%UnverifiedDeepSeek-V2 adopted a configuration of 1 shared expert plus multiple routed experts, achieving results superior to pure routed MoE.
50%UnverifiedThe activation rate of Mixtral 8x7B is approximately 25%, and the activation rate of DeepSeek-V2 is approximately 10%.
50%UnverifiedDeepSeek-V2将每个Transformer层的FFN拆分为多个专家模块，通过门控网络动态路由，每个token只激活约10%的参数
50%