MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
The theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
Timeline (last 90 days)
Mistral Large 4 uses a MoE architecture with 1T total parameters and 52B activated parameters.
In a MoE architecture, the total parameters determine the hardware threshold (VRAM must hold all weights), while the activated parameters determine the running speed.
Selection based on internal expert activations requires a precomputation pass, while feeding multiple full representations separately into the same expert repeats that expert's computation, and under a fixed expert budget both approaches erode the efficiency advantage of MoE.
The consistency between the ranking of a single router and the ranking from actual expert utility evaluation is weak, especially in the middle and deep layers.
Both models use a mixture-of-experts (MoE) and hybrid Mamba-Transformer architecture, targeting reasoning, long-context understanding and agentic workloads.
Model miniaturization techniques include MoE sparse architectures, parameter pruning, and others.
In MoE training, the cost of moving intermediate results is often on the same order as the computation itself, and fused kernels can amplify the gains.
Qwen 3.6 35B uses a MoE architecture, and when all expert weights are placed on the CPU with only a small amount of weights kept on the GPU, it can maintain a generation speed of about 40 tokens/second.
In a MoE architecture, all parameters still need to be loaded into memory, which is the source of the 111GB size.
The number of unique experts should be counted per MoE layer rather than as a global average, because routing distributions can differ greatly across layers and a global average would mask extreme union cases.
40 more timeline events
All Facts (20)
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
90%VerifiedFlash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedMuse Spark Open uses an MoE architecture, with 816B total parameters and 42B activated parameters.
85%VerifiedMistral Large 4 adopts a sparse Mixture of Experts (MoE) architecture, with 49 billion (49B) parameters actually activated during each inference.
85%VerifiedQwen3.8-Flash-Next adopts a Mixture of Experts (MoE) architecture with 125B total parameters and 6B activated parameters.
85%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%VerifiedDeepSeek V4 Flash is a cost-effective inference model released by DeepSeek, optimized for speed and cost.
80%