Layer Normalization
A neural network normalization technique that standardizes activations to zero mean and unit variance across the feature dimension for each sample. It stabilizes activation values across layers and is widely used to improve training stability in Transformer and other sequence models.
Timeline (last 90 days)
Residual connections mitigate the vanishing gradient problem in deep networks, and layer normalization normalizes each token's vector (mean 0, variance 1) to stabilize training dynamics.
Layer normalization standardizes each sample across the feature dimension to zero mean and unit variance
All Facts (4)
Residual connections mitigate the vanishing gradient problem in deep networks, and layer normalization normalizes each token's vector (mean 0, variance 1) to stabilize training dynamics.
50%UnverifiedLayer normalization standardizes each sample across the feature dimension to zero mean and unit variance
50%UnverifiedGPT-2引入了Pre-LN（前置层归一化）设计，相比原始Transformer的Post-LN，将LayerNorm置于残差块内部的子层之前，改善了深层网络的梯度流动稳定性
50%Unverified残差连接是 Transformer 的核心创新之一，每一层的输出是输入加上该层的变换结果
50%