Noise Residual
The noise residual is the remaining part of the MTP gradient that is orthogonal to the KL self-distillation signal. It does not correspond to a beneficial learning objective and interferes with the standard next-token prediction loss. It may originate from the accumulated error of multi-step prediction and from gradient conflicts.
Timeline (last 90 days)
The gradient of standard MTP contains two components: a KL self-distillation signal and a noise residual.
The noise residual harms the performance of next-token prediction.
Soft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
All Facts (3)
The gradient of standard MTP contains two components: a KL self-distillation signal and a noise residual.
50%UnverifiedThe noise residual harms the performance of next-token prediction.
50%UnverifiedSoft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
50%