SwiGLU

The developer manually implemented four core LLaMa architecture components—RoPE, GQA, RMSNorm, and SwiGLU—without relying on an existing training framework.

The SwiGLU activation function uses a Swish variant plus a gated linear unit, and typically brings better language modeling results at an equal parameter count.

SwiGLU is a commonly used gated activation in Transformers, of the form SiLU(xW1) ⊗ (xW3), where SiLU contains an exponential operation and tends to lose precision under BF16.

The study conducted experiments on four Qwen series models, and the results show that simultaneously aligning the three regions of RoPE, SwiGLU, and residual RMSNorm can completely prevent training collapse under the default BF16 baseline.

The method proposed in the study keeps selected intermediate results at FP32 precision in three key regions—RoPE (rotary position embedding), SwiGLU (activation function), and residual RMSNorm—and only rounds at their BF16 output stage.

The SwiGLU double-negative loophole: when the gate branch and the value branch output negative values at the same time, the product is positive, producing spurious positive activations that are misread as polysemanticity; after fixing this mathematical loophole, the neuron returns to a single function.

On the gate/up projection with SwiGLU at 8B shapes, FuTransformer's BF16 path is 1.13x faster than CODA, and its FP8 path is 2.02 to 2.18x faster than the evaluated BF16 CODA path.

RoPE, SwiGLU and GQA are structural components widely adopted by current mainstream large models (including the Qwen series).

Kernel fusion techniques can merge multiple consecutive operations into a single kernel, for example fusing RoPE with attention computation or fusing SwiGLU's matrix multiplication with activation, thereby reducing memory round trips and scheduling overhead.

In MoE architectures, the number of gating activation calls grows linearly with the number of experts, causing the proportion of SwiGLU/SiLU computation to be far higher than in dense models.

On the NVIDIA GB200 platform, full training step throughput improves by 8.0% for routed-expert SwiGLU tasks, the highest gain among the four tasks.

Different models differ in attention mechanisms, activation functions, and normalization methods (e.g., GQA, RoPE, SwiGLU), and each variant requires the runtime framework to provide corresponding operator implementations.

Agnes-3.0-Flash's feed-forward network uses SwiGLU with an intermediate dimension of 17408, with a parallel 2048-dimensional SwiGLU branch per layer

Noam Shazeer's SwiGLU activation function paper attributes the architecture's success to "divine benevolence" in the conclusion section

The model architecture is a cross-attention DiT (896 width × 16 layers) using 2D RoPE, QK-norm, SwiGLU, and adaLN-single.

The FFN feed-forward network module upscales vectors via the up layer (e.g., 4K→8K), passes them through an activation function (such as SwiGLU gated activation), and then reduces them back to the original dimension via the down layer.

The SwiGLU activation function is the feed-forward network design adopted by mainstream modern large models such as LLaMA.

SwiGLU increases the number of feed-forward layer parameters by approximately 50%, but has been proven to consistently improve model quality under the same compute budget, and is adopted by both Meta's LLaMA and Google's PaLM.

SwiGLU was proposed by Google researcher Noam Shazeer in 2020, combining the Swish activation function with GLU.