Gradient Flow
A mathematical framework that interprets neural network training as a continuous dynamical system. It represents the limit of discrete gradient descent as the learning rate approaches zero in the form of the ordinary differential equation dθ/dt = −∇L(θ), providing a theoretical tool for analyzing training dynamics.
Timeline (last 90 days)
For finite step sizes, even if a stationary point is unstable under continuous-time gradient flow, if the step size is too large it may skip over the escape direction and cause numerical divergence, while if the step size is too small the escape time may grow exponentially and appear as stagnation.
The paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
The paper analyzes two extended settings: more general stopping times and target functions supported on multiple spherical harmonic orders
Standard backpropagation is essentially a discrete iteration of gradient descent, and when the learning rate is regarded as an infinitesimally small time step, this discrete process converges to the gradient flow ordinary differential equation dθ/dt = −∇L(θ).
All Facts (4)
For finite step sizes, even if a stationary point is unstable under continuous-time gradient flow, if the step size is too large it may skip over the escape direction and cause numerical divergence, while if the step size is too small the escape time may grow exponentially and appear as stagnation.
50%UnverifiedThe paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
50%UnverifiedThe paper analyzes two extended settings: more general stopping times and target functions supported on multiple spherical harmonic orders
50%UnverifiedStandard backpropagation is essentially a discrete iteration of gradient descent, and when the learning rate is regarded as an infinitesimally small time step, this discrete process converges to the gradient flow ordinary differential equation dθ/dt = −∇L(θ).
50%