AdamW

AdamW corrects the confusion between weight decay and L2 regularization in Adam by decoupling weight decay from the gradient update, and has become the de facto standard optimizer for training Transformers.

The model weights alone of a 175-billion-parameter model (in BF16 precision) require 350 GB of storage, and the Adam optimizer must maintain two sets of states—first-order moments and second-order moments—totaling more than 1 TB.

DeepSeek V4采用MUON优化器替代AdamW进行训练

Decoupled DiLoCo的外部优化器通常使用Nesterov动量SGD，内部优化器通常为AdamW

MUON优化器替代了沿用多年的AdamW优化器，通过引入矩阵正交化步骤使训练曲线更加平滑

ApexW is currently better seen as an experimental project worth tracking rather than a production-environment replacement for AdamW.

AdamW needs to maintain two sets of moving statistics (first moment and second moment) with the same shape as the parameters for each trainable parameter, usually stored in FP32.

Starting from 2.x, PyTorch provides an AdamW implementation with fused=True, which completes parameter updates with a single CUDA kernel and is already the official optimization path.

General-purpose optimizers such as SGD and AdamW are not designed for single-pass, non-stationary data distributions, and their update rules cannot effectively balance the needs of rapid adaptation and resistance to forgetting.

Llama 2 used the AdamW optimizer (beta1 = 0.9, beta2 = 0.95, weight decay 0.1, gradient clipping 1.0), and was pretrained on 2 trillion tokens.

On the Qwen2.5-32B model, merely increasing AdamW's first-order momentum coefficient β₁ from its conventional value brought nearly a 2 percentage point improvement in average validation accuracy, without modifying other hyperparameters.

In the RLVR experiments on Qwen3-4B-Base, with both the SGD+EMA and AdamW optimizers, configurations choosing a larger momentum coefficient showed an advantage.

AdamW's default value of β₁ is usually 0.9, controlling the exponential moving average window of the first-order gradient.

AdamW's effective step size is approximately proportional to η/(1-β₁), so simply increasing β₁ amounts to implicitly amplifying the effective learning rate.

Compared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.

AdamW maintains two exponential moving average states, the first moment and the second moment, for each trainable parameter, usually stored in fp32, and in mixed-precision training the optimizer states often take up more GPU memory than the model weights and activations.

In AdamW, the parameter update is approximately equal to the first moment divided by (the square root of the second moment plus ε), and the reciprocal of the square root of the second moment acts as a preconditioner, adaptively scaling the gradient for each coordinate.

The author's fine-tuning used the QLoRA method, with LoRA rank/alpha of 32/64, a learning rate of 1e-4, batch size 1 with gradient accumulation of 16, AdamW 8-bit optimizer, and training for 1 epoch.

The weight decay simplification idea in this work shares common ground with the decoupled weight decay approach in AdamW.

In the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.