Softmax Collapse
A numerical defect in standard softmax implementations: when the target logit is far larger than the other logits, the gradient becomes zero due to floating-point rounding, disrupting gradient balance.
Timeline (last 90 days)
The standard softmax implementation has two numerical bugs: Softmax Collapse and Softmax Cancellation.
Softmax Collapse causes the gradient of the target logit to be rounded to zero, breaking the property that gradients sum to zero.
The gradient imbalance caused by Softmax Collapse may lock the model prematurely into a memorization solution, delaying or preventing the transition to a generalization solution.
All Facts (3)
The standard softmax implementation has two numerical bugs: Softmax Collapse and Softmax Cancellation.
50%UnverifiedSoftmax Collapse causes the gradient of the target logit to be rounded to zero, breaking the property that gradients sum to zero.
50%UnverifiedThe gradient imbalance caused by Softmax Collapse may lock the model prematurely into a memorization solution, delaying or preventing the transition to a generalization solution.
50%