Switch Transformer

Google's Switch Transformer, released in 2021, applied MoE at scale to Transformer language models, expanding model capacity to the trillion-parameter level.

Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.

The large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.

The Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.

The core idea of MoE (Mixture of Experts) can be traced back to the early mixture of experts framework proposed by Jacobs et al. in 1991, and Google's Switch Transformer in 2021 brought it into the era of large-scale language models.

The MoE architecture was first proposed by Jacobs et al. in 1991 and in recent years has been brought back into the mainstream by models such as Google's Switch Transformer and Mixtral.

MoE was reactivated in the era of large language models by Google's Switch Transformer (2021) and GLaM.

Google validated the Mixture of Experts (MoE) architecture at scale in Switch Transformer.

The modern form of MoE can be traced back to a Google paper in 2017, and in 2022 Google's Switch Transformer demonstrated that trillion-parameter sparse models can be trained efficiently.

Switch Transformer and the Mixtral series have widely adopted the MoE architecture.

The mixture-of-experts architecture can be traced back to research by Jacobs et al. in 1991; Google's Switch Transformer (2021) and Mistral's Mixtral 8x7B (2023) brought it back into the mainstream.

Switch Transformer, proposed in 2021, simplified the number of experts routed to per token to 1.

Google's Switch Transformer and GLaM were the first to scale MoE to the trillion-parameter level.

The most common form of load balancing loss is the auxiliary loss, represented by Switch Transformer, with the formula L_balance = α · N · Σ(f_i · p_i), where N is the number of experts and α is the loss coefficient.

The idea of MoE sparse activation was first validated in natural language processing by Switch Transformer, GShard, and others.

Switch Transformer and Mixtral 8x7B are typical representatives of the MoE architecture, with a large total parameter count but only a subset of experts activated each time.

Entropy regularization techniques have been widely applied in the MoE domain, such as the load balancing loss in GShard and Switch Transformer.

Sparse activation routing mechanisms have been validated in research such as Switch Transformer and Mixtral, where input tokens are dynamically assigned to the most relevant expert sub-networks via a gating network.

The core advantage of the MoE (Sparse Mixture of Experts) architecture is that it significantly reduces the computational cost of a single inference through sparse activation while maintaining a large parameter count; the architectural concept can be traced back to research by Jacobs et al. in 1991

The Mixture of Experts architecture can be traced back to the theoretical framework proposed by Jacobs et al. in 1991, but it was not truly introduced into the field of large-scale language models until Google's Switch Transformer paper in 2017.