MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
GPT-4 uses a Mixture of Experts (MoE) architecture, with an estimated total parameter count of 1.8 trillion, and approximately 1/8 of the total parameters are actually activated during each inference.
Timeline (last 90 days)
Chapter 3 breaks down the MoE layer, analyzing the router, routing algorithms, load balancing, experts, block layout, training process, and system implementation one by one.
In MoE architectures, a model with 671B total parameters may activate only about 37B parameters per inference.
MoE, hybrid layers, and ternary weights can be used in combination, and each also stands on its own.
One reason the author chose MoE as the experimental target is that the erroneous behavior may involve only specific experts or routing paths, making localized intervention theoretically more feasible.
A 65B dense model has higher inference-time VRAM usage and throughput costs than a MoE model of the same parameter count, and can serve fewer concurrent requests on the same hardware.
Ripple-65B is a dense model that does not use a MoE architecture, and all parameters are activated on every inference.
MoE's experts are sub-networks that are trained, and the routing is end-to-end differentiable.
MAEP is a training-free expert pruning framework that requires only a single forward pass to decompose expert importance by modality, compressing a roughly 58 GB MoE multimodal large model into a single 40 GB GPU.
In MoE multimodal large models, expert parameters account for more than 90% of the model's size.
If the goal is high-concurrency, low-latency online serving, MoE usually has the advantage in cost-effectiveness.
40 more timeline events
All Facts (20)
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
90%VerifiedFlash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedGPT-4 uses a Mixture of Experts (MoE) architecture, with an estimated total parameter count of 1.8 trillion, and approximately 1/8 of the total parameters are actually activated during each inference.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedMuse Spark Open uses an MoE architecture, with 816B total parameters and 42B activated parameters.
85%VerifiedMistral Large 4 adopts a sparse Mixture of Experts (MoE) architecture, with 49 billion (49B) parameters actually activated during each inference.
85%VerifiedQwen3.8-Flash-Next adopts a Mixture of Experts (MoE) architecture with 125B total parameters and 6B activated parameters.
85%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedIf a MoE model activates only approximately 1/8 of its parameters per inference, the actual computational overhead may be equivalent to that of a dense model with tens of billions of parameters.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%