Adam
Adam (Adaptive Moment Estimation) is an adaptive learning rate optimization algorithm for training deep learning models, proposed by Diederik Kingma and Jimmy Ba. It combines the ideas of momentum and RMSProp by simultaneously estimating the first moment (mean) and second moment (variance) of gradients to automatically adjust the learning rate for each parameter. Adam is characterized by computational efficiency, minimal hyperparameter tuning requirements, and fast convergence, and is widely applied in neural network training tasks.
Core Facts
Adam combines the momentum method with RMSprop's adaptive learning rate, independently maintaining exponential moving averages of the first-order and second-order gradients for each parameter.
The Adam optimizer was proposed in 2014, combining momentum and adaptive learning rates, and has become the most widely used optimizer today.
AdamW corrects the confusion between weight decay and L2 regularization in Adam by decoupling weight decay from the gradient update, and has become the de facto standard optimizer for training Transformers.
Timeline (last 90 days)
UTA does not depend on the specific dynamics of SGD, can be seamlessly integrated into modern optimizers such as Adam, and is plug-and-play with negligible extra overhead.
The derivation of UBA is built on the continuous-time dynamics of SGD, so its optimality conclusion holds for SGD but is difficult to transfer directly to adaptive optimizers such as Adam.
After the model memorizes the training data, logit growth is constrained by the balance between weight decay and the ε hyperparameter in the Adam optimizer.
Deep learning parameter dimensions are extremely high, making exact second-order methods infeasible; in practice, quasi-Newton methods (such as L-BFGS) or adaptive methods (such as Adam) are mostly used to exploit second-order information.
DP-Adam is a privacy version that embeds the clipping-and-noising step of DP-SGD into the momentum estimation of the Adam optimizer; DP-Muon is a privacy version that uses the Muon optimizer (based on matrix orthogonalization updates) for matrix parameters.
Bias correction is one of the key designs that distinguishes Adam from RMSProp.
Adam's bias correction is performed by dividing the first and second moments by (1-β1^t) and (1-β2^t) respectively, where t is the time step and β1 and β2 are the decay rates.
Adam maintains two exponential moving averages: the first moment of the gradient (momentum) and the second moment (squared gradient).
The video builds the optimizer step by step in the order of plain GD, Nesterov, AdaGrad, RMSProp, and Adam.
GPT-3 training used the Adam optimizer (beta1 = 0.9, beta2 = 0.95), gradient clipping to 1.0, a learning rate warmed up over 375 million tokens then cosine-decayed to 10% of its peak over 260 billion tokens, and weight decay of 0.1.
38 more timeline events
All Facts (20)
Adam combines the momentum method with RMSprop's adaptive learning rate, independently maintaining exponential moving averages of the first-order and second-order gradients for each parameter.
80%VerifiedThe Adam optimizer was proposed in 2014, combining momentum and adaptive learning rates, and has become the most widely used optimizer today.
80%VerifiedAdamW corrects the confusion between weight decay and L2 regularization in Adam by decoupling weight decay from the gradient update, and has become the de facto standard optimizer for training Transformers.
75%VerifiedThe model weights alone of a 175-billion-parameter model (in BF16 precision) require 350 GB of storage, and the Adam optimizer must maintain two sets of states—first-order moments and second-order moments—totaling more than 1 TB.
75%VerifiedGradient descent has spawned three main variants — batch gradient descent, stochastic gradient descent, and mini-batch gradient descent — and the Adam optimizer is a combined improvement built on momentum and adaptive learning rates.
65%VerifiedUsing the Adam optimizer with a default learning rate of 1e-3 as a starting point is a robust entry strategy for deep learning.
65%VerifiedAdaptive learning rate optimizers such as Adam and RMSProp further improve convergence speed and stability over SGD.
65%UnverifiedAdam的一阶矩和二阶矩的计算都基于指数移动平均（EMA）
95%UnverifiedAdam的论文标题为《Adam: A Method for Stochastic Optimization》
95%UnverifiedAdam由Diederik P. Kingma和Jimmy Ba于2014年提出，发表于ICLR 2015
95%UnverifiedAdam的二阶矩估计记录梯度平方的指数移动平均，反映梯度的波动幅度
90%UnverifiedAdam的EMA设计只需要存储上一时刻的矩估计值，内存开销极小
90%UnverifiedAdam在RMSProp基础上进一步引入一阶矩（动量），并增加了偏差修正机制
90%UnverifiedOpen Code最初叫Terminal AI Agent，由一位开发者业余时间创建，后来知名开发者Dex和Adam加入
85%UnverifiedAdam可以被理解为AdaGrad和RMSProp的进化版本
80%UnverifiedWhen using the Adam optimizer, 16 bytes of state information are required per parameter (4 bytes for FP32 parameters + 4 bytes for FP32 gradients + 4 bytes for first-order momentum + 4 bytes for second-order momentum), requiring at least approximately 24 GB of state information per GPU for a 1.5 billion parameter model.
60%UnverifiedAdam (Adaptive Moment Estimation) is one of the most widely used optimizers in deep learning.
60%UnverifiedAdam的一阶矩衰减系数β₁通常默认为0.9
55%UnverifiedThe derivation of UBA is built on the continuous-time dynamics of SGD, so its optimality conclusion holds for SGD but is difficult to transfer directly to adaptive optimizers such as Adam.
50%UnverifiedUTA does not depend on the specific dynamics of SGD, can be seamlessly integrated into modern optimizers such as Adam, and is plug-and-play with negligible extra overhead.
50%