Mixtral
An open-source large language model based on the Mixture-of-Experts (MoE) architecture, released by Mistral AI. It brought MoE architecture into mainstream community awareness through its use of sparse activation mechanisms.
Core Facts
Mistral is a French AI company known for its efficient open-source models.
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
The MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
Timeline (last 90 days)
MoE is widely used in large-scale language models (such as Mixtral, DeepSeek, etc.) and is an important means of scaling model capacity without linearly increasing inference overhead.
Mistral built community reputation through open-weight models such as Mistral 7B and Mixtral, while the Large series is the pillar of its commercialization revenue.
Mixture-of-Experts (MoE) is the most mature engineering implementation of the modular idea in deep learning, whose core is setting up multiple expert subnetworks and a gating network in the network.
The MoE technical approach was later widely adopted by models such as DeepSeek and Mixtral, becoming a mainstream solution for reducing costs and improving efficiency in large models.
Mixtral is a portmanteau of Mixture-of-experts and Mistral, referring to a sparse mixture-of-experts model.
MoE models such as DeepSeek and Mixtral adopt the "large total, small activation" naming convention.
Nemotron Bolt 3.5's competitors in the local MoE deployment track include Mistral's Mixtral series, Qwen's MoE variants, and DeepSeek's lightweight models.
Mistral's Mixtral series adopts a sparse expert architecture, has a clear advantage in inference cost, and its efficiency-oriented technical路线 is naturally aligned with the power constraints of on-orbit deployment.
Groq is a platform providing inference acceleration services, using its self-developed LPU chips to let open-source large models return results with extremely low latency, and it provides an OpenAI-compatible API.
Groq itself is not a model, but a platform that provides access to multiple open-source models (such as Llama, Qwen, Mixtral, etc.), allowing the model to be switched at any time without changing the workflow structure.
25 more timeline events
All Facts (20)
Mistral is a French AI company known for its efficient open-source models.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedThe MoE architecture was first proposed by Jacobs et al. in 1991 and in recent years has been brought back into the mainstream by models such as Google's Switch Transformer and Mixtral.
80%VerifiedTogether AI is a cloud inference platform focused on open-source large models, supporting mainstream open-source models such as Llama, Mixtral, and Qwen.
75%VerifiedMeta open-sourced the Llama series of models, Alibaba open-sourced the Qwen series, and Mistral AI has also adopted an open-source strategy.
75%VerifiedGroq hosts open-source models such as Llama and Mixtral via API and provides a free quota.
65%VerifiedThe Mixture of Experts (MoE) architecture activates only the most relevant expert sub-networks through a routing mechanism (typically 1/8 to 1/4 of total parameters); both Mistral's Mixtral and Google's Gemini 1.5 adopt this architecture.
65%VerifiedMeta's LLaMA series, Google's Gemma, Mistral AI's Mistral/Mixtral, and Alibaba's Qwen series are all supported in Transformers.
65%UnverifiedMixtral and DeepSeek-V2 adopt sparse activation strategies
90%UnverifiedMeta 的 LLaMA 系列、Mistral AI 的 Mistral/Mixtral、Google 的 Gemma 等开源模型都选择通过 Hugging Face 平台和 Transformers 格式进行发布
90%UnverifiedMoE架构最早由Google在2017年提出，后被Mistral的Mixtral系列和阿里的Qwen系列等广泛采用
85%UnverifiedDBRX在发布时在多项基准测试中超越了同期的Llama 2 70B和Mixtral等开源模型
75%UnverifiedSwitch Transformer and the Mixtral series have widely adopted the MoE architecture.
70%UnverifiedSGLang对MoE模型的专家并行支持比vLLM更为成熟，使其成为部署Mixtral、DeepSeek、Kimi K2.6等MoE架构模型的首选框架之一
70%UnverifiedThe MoE architecture decouples parameter scale from computational cost through sparse activation, and models such as Mixtral and DeepSeek have validated its feasibility in practice.
60%