SGD

The Adam optimizer was proposed in 2014, combining momentum and adaptive learning rates, and has become the most widely used optimizer today.

Stochastic gradient descent (SGD) uses only one sample at a time, which is fast but produces noisy update directions.

Gradient descent has three variants: batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent.

Adaptive learning rate optimizers such as Adam and RMSProp further improve convergence speed and stability over SGD.

Mini-batch gradient descent (Mini-batch GD) is a compromise between batch gradient descent and stochastic gradient descent, and is also the most commonly used gradient descent method in industry.

The derivation of UBA is built on the continuous-time dynamics of SGD, so its optimality conclusion holds for SGD but is difficult to transfer directly to adaptive optimizers such as Adam.

General-purpose optimizers such as SGD and AdamW are not designed for single-pass, non-stationary data distributions, and their update rules cannot effectively balance the needs of rapid adaptation and resistance to forgetting.

For stochastic gradient descent (SGD), the framework either recovers or significantly sharpens all existing bounds known to the authors.

Under infinite-variance gradients, the convergence rate of SGD is determined by the tail index t rather than the conventional 1/sqrt(n) rate.

The paper's results apply to observation horizons with state-dependent noise and growth, and for any usable gradient moment p>1, the admissible window is non-empty.

In the infinite-variance case t ∈ (1,2), the central fluctuation scale of SGD is of order η^{1/t}, and the limit is a t-stable distribution.

The paper derives explicit projected and excess-objective tails, extreme quantiles, and, for a fixed batch size b, the factor b^{1-t}.

The paper validates the theoretical results through coupled simulations from one to one hundred dimensions, showing that the theoretically predicted tail probabilities agree well with the actual simulations.

The paper's explicit tail expressions can be used to design more robust optimization algorithms, such as adaptive step sizes or gradient clipping strategies.

The paper proves that the SGD iterates converge to a stable OU process at the central fluctuation scale and uses this limit to characterize the tail probabilities.

The paper's main result is that for regularly varying innovations (index t ∈ (1,2)), the tail probability of the last iterate is dominated by a single large gradient shock transported through the local Hessian flow.

Optimizers such as stochastic gradient descent (SGD), the momentum method, and Adam are all variants that improve the update strategy on the basic gradient descent framework.

When the step size $\eta \to 0$, the trajectory of the SGD iteration $x_{k+1} = x_k - \eta \nabla f(x_k) + \eta \xi_k$ can be approximated by an SDE: $dX_t = -\nabla f(X_t)\,dt + \sqrt{\eta}\,\Sigma\,dW_t$.

The rate transfer theorem states that if the Lyapunov generator inequality holds for the coefficients $(b,\sigma)$, and the SGD update operator and the SDE coefficients satisfy certain moment and smoothness matching conditions, then the expected Lyapunov function of the discrete iteration also decays at a similar rate, and an explicit finite-time bound is given.

Traditional SGD analyses treat momentum as an auxiliary variable or simply drop it, ignoring the geometrically decaying memory that the momentum state itself carries as a recursive variable.