Gradient Descent

An excessively large learning rate in gradient descent causes parameters to oscillate or even diverge around the optimal solution, while an excessively small rate results in extremely slow convergence.

Gradient descent iteratively adjusts weights in the opposite direction of the loss function's gradient, causing the prediction error to converge progressively.

The essence of backpropagation is chain-rule differentiation, where the gradient of the loss function with respect to the parameters of each layer is propagated layer by layer from the output layer to the input layer, on the premise that the loss function must exist and be differentiable.

The core process of training a model is to iteratively update parameters in the direction of the loss function's gradient via gradient descent.

Catastrophic forgetting is a classic problem in the field of neural networks; as early as the 1990s, researchers discovered that the performance on old tasks drops sharply when neural networks are continuously trained on new tasks

Gradient descent updates parameters through the iterative formula θ = θ - α∇J(θ), where α is the learning rate and ∇J(θ) is the vector of partial derivatives of the loss function with respect to the parameters.

Gradient descent is the most fundamental optimization algorithm in machine learning; a learning rate that is too large causes oscillation, while one that is too small results in slow convergence.

Machine learning measures the gap between predicted values and true labels through a loss function, then uses optimization algorithms such as gradient descent to gradually adjust model parameters to reduce the error.

Gradient descent has three variants: batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent.

Gradient descent is the most fundamental optimization algorithm in deep learning, and its mathematical essence is iteratively updating parameters in the direction of the negative gradient of the loss function.

The learning rate determines how large each step is: too large causes overshooting the valley bottom and divergence, while too small makes training extremely time-consuming.

Gradient descent has spawned three main variants — batch gradient descent, stochastic gradient descent, and mini-batch gradient descent — and the Adam optimizer is a combined improvement built on momentum and adaptive learning rates.

The universal approximation theorem is a purely existential conclusion that guarantees the existence of an approximation scheme but provides no tight upper bound on the number of neurons required.

The parameter update rule for gradient descent is θ_{t+1} = θ_t - η·∇L(θ_t), where η is the learning rate, and updating parameters in the direction of the negative gradient reduces the loss function most rapidly.

Gradient descent mathematically involves partial derivatives and the chain rule, and its intuitive essence is a 'downhill' process of finding the lowest point.

In neural networks, parameters jointly encode knowledge of multiple tasks through gradient descent; when the model trains on a new task, gradient updates substantially change the weights critical to old tasks, while old task data no longer appears, causing the model to gradually forget the old tasks.

Traditional learning algorithm analysis usually assumes that accurate gradient information is available at every moment and at every synapse.

The algorithm improves the estimates of the hidden weight vectors through gradient-descent-like updates on the Euclidean sphere.

First-order Taylor expansion corresponds to the linear approximation of gradient descent, while second-order expansion introduces curvature information and corresponds to second-order optimization methods such as Newton's method.

Constructing a bump by subtracting two Sigmoids is consistent with the classic signal processing idea of obtaining a rectangular window by subtracting two Heaviside step functions, but the Sigmoid provides a smooth differentiable version, making gradient-based training possible.