DeepSeek-MoE
A sparse mixture-of-experts (MoE) large language model released by DeepSeek, employing innovations such as fine-grained expert segmentation and shared expert isolation. It has approximately 145B total parameters and approximately 22B activated parameters.
Timeline (last 90 days)
The MoE architecture uses a lightweight router to decide which experts handle each input token, and in mainstream implementations each token typically activates only 2 to 8 experts.
K=2 in MoE is currently the most common choice in practice, and models including DeepSeek-MoE and Mixtral 8x7B both adopt this setting.
All Facts (2)
The MoE architecture uses a lightweight router to decide which experts handle each input token, and in mainstream implementations each token typically activates only 2 to 8 experts.
50%UnverifiedK=2 in MoE is currently the most common choice in practice, and models including DeepSeek-MoE and Mixtral 8x7B both adopt this setting.
50%