Backpropagation

Reverse-mode automatic differentiation in neural network training is the mathematical essence of backpropagation, because the number of parameters far exceeds the output dimensionality of the loss value.

The mathematical foundation of the backpropagation algorithm is the chain rule in calculus, which decomposes the gradient of a composite function into the product of local gradients at each layer, propagating the error signal layer by layer from the output layer to the input layer.

The backpropagation algorithm was popularized by Rumelhart, Hinton, and Williams in their 1986 paper, with the chain rule as its mathematical core.

Backpropagation is the mathematical application of the chain rule.

The essence of backpropagation is chain-rule differentiation, where the gradient of the loss function with respect to the parameters of each layer is propagated layer by layer from the output layer to the input layer, on the premise that the loss function must exist and be differentiable.

The core idea of backpropagation is to use the chain rule to decompose output error layer by layer, allowing the error signal to flow backward from the output layer to every weight parameter.

Backpropagation is essentially a dynamic programming algorithm for efficiently computing gradients, which reduces the computational complexity of gradient calculation by propagating error signals layer by layer from the output layer to the input layer and reusing intermediate computation results.

反向传播算法的本质是通过链式法则在计算图上高效计算标量损失对每个参数的偏导数

Backpropagation is nearly impossible in biological brains because it requires the use of symmetric weights in the forward and backward paths (the weight transport problem), and no known mechanism in real brains supports this.

Backpropagation is essentially a systematic application of the chain rule, which efficiently computes the partial derivative of each parameter with respect to the loss function by propagating error signals layer by layer from the output layer to the input layer.

Traditional learning algorithm analysis usually assumes that accurate gradient information is available at every moment and at every synapse.

In backpropagation, the output layer error term formula is δ_out = (a_out - y) * σ'(z_out).

The hidden layer error term formula is δ_hidden = (W_out^T · δ_out) * σ'(z_hidden).

The weight gradient formula is ∂L/∂W = δ · a_prev^T.

The formula δ_out = (a_out - y) * σ'(z_out) in the text omits the constant factor 2, which in practice does not affect the direction of gradient descent and is merely equivalent to a scaling of the learning rate.

Backpropagation's credit assignment solution is based on the chain rule, computing sensitivity layer by layer from the output end, and can obtain the gradient of each weight at a cost of roughly one forward pass.

In the Handwired network, the partial derivative of the shipping fee with respect to the per-kilogram rate is 2, with respect to the mid-tier price is 1, and with respect to all thresholds ($50 membership line, $100 line, 500 grams, 2 kilograms) is 0.

Backpropagation is the core mechanism of neural network training and is the underlying algorithmic foundation of NLP, image recognition, and large language models.

Backpropagation reuses the gradient computation results of intermediate layers, reducing the computational complexity of taking derivatives with respect to parameters from exponential to linear in relation to network size.

The optimization targets of inference-specific NPUs usually include quantized arithmetic, fixed-function data paths, a highly constrained memory hierarchy, and low-power, low-latency forward inference, rather than optimization for backpropagation.