Concept虚假梯度Bug
Softmax Cancellation
A numerical defect in standard softmax implementations: in knowledge distillation, when the student and teacher logits are equal, separately normalizing and then subtracting produces a nonzero spurious gradient, causing the student to drift.
Timeline (last 90 days)
Oct 23
The standard softmax implementation has two numerical bugs: Softmax Collapse and Softmax Cancellation.
Unverified50%
Oct 23
In knowledge distillation, when the student logits exactly equal the teacher logits, the standard softmax implementation produces non-zero spurious gradients.
Unverified50%