RMSNorm
A normalization method that scales only by the root mean square, without mean subtraction or bias addition; compared with LayerNorm it is computationally simpler while remaining stable in training.
Timeline (last 90 days)
The developer manually implemented four core LLaMa architecture components—RoPE, GQA, RMSNorm, and SwiGLU—without relying on an existing training framework.
RMSNorm replaces LayerNorm with root-mean-square normalization, making computation simpler and numerical stability during training better.
The method proposed in the study keeps selected intermediate results at FP32 precision in three key regions—RoPE (rotary position embedding), SwiGLU (activation function), and residual RMSNorm—and only rounds at their BF16 output stage.
Residual RMSNorm performs root-mean-square normalization on the residual branch, and the normalization coefficient is determined by global statistics, so any premature rounding is amplified across the entire hidden state.
The study conducted experiments on four Qwen series models, and the results show that simultaneously aligning the three regions of RoPE, SwiGLU, and residual RMSNorm can completely prevent training collapse under the default BF16 baseline.
FlyDSL covers core operations such as dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm and TopK.
The standard Transformer (Post-LN) performs normalization after residual addition, while many modern large models such as LLaMA switch to Pre-LN or RMSNorm, moving normalization to before the residual addition.
RMSNorm replaces LayerNorm, reducing computational overhead while maintaining effectiveness.
RMSNorm 相比传统 LayerNorm 去掉了均值中心化步骤，仅保留均方根归一化，计算更轻量
Gemma 采用旋转位置编码（RoPE）、GeGLU 激活函数、RMSNorm 归一化等现代 Transformer 设计
All Facts (10)
The developer manually implemented four core LLaMa architecture components—RoPE, GQA, RMSNorm, and SwiGLU—without relying on an existing training framework.
50%UnverifiedRMSNorm replaces LayerNorm with root-mean-square normalization, making computation simpler and numerical stability during training better.
50%UnverifiedThe method proposed in the study keeps selected intermediate results at FP32 precision in three key regions—RoPE (rotary position embedding), SwiGLU (activation function), and residual RMSNorm—and only rounds at their BF16 output stage.
50%UnverifiedResidual RMSNorm performs root-mean-square normalization on the residual branch, and the normalization coefficient is determined by global statistics, so any premature rounding is amplified across the entire hidden state.
50%UnverifiedThe study conducted experiments on four Qwen series models, and the results show that simultaneously aligning the three regions of RoPE, SwiGLU, and residual RMSNorm can completely prevent training collapse under the default BF16 baseline.
50%UnverifiedFlyDSL covers core operations such as dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm and TopK.
50%UnverifiedThe standard Transformer (Post-LN) performs normalization after residual addition, while many modern large models such as LLaMA switch to Pre-LN or RMSNorm, moving normalization to before the residual addition.
50%UnverifiedRMSNorm replaces LayerNorm, reducing computational overhead while maintaining effectiveness.
50%UnverifiedRMSNorm 相比传统 LayerNorm 去掉了均值中心化步骤，仅保留均方根归一化，计算更轻量
50%UnverifiedGemma 采用旋转位置编码（RoPE）、GeGLU 激活函数、RMSNorm 归一化等现代 Transformer 设计
50%