Soft-MTP
A method that improves Multi-Token Prediction (MTP) training by modifying the residual connection of hidden states (h_ntp += h_mtp.detach() - h_mtp), decoupling the KL self-distillation signal from noisy residuals so that gradients flow only through the beneficial KL self-distillation component, thereby improving cross-task performance without additional computational overhead.
Timeline (last 90 days)
Soft-MTP can separate the KL self-distillation gradient with a one-line code change (h_ntp += h_mtp.detach() - h_mtp).
Soft-MTP does not require redesigning the model architecture or an additional training stage, and the existing MTP pretraining pipeline can be applied directly.
Soft-MTP improves MTP's performance across a variety of tasks.
The benefits of Soft-MTP depend on the premise that the MTP head and the NTP head share the backbone network.
The change in gradient routing may affect training stability, especially when the loss surface is steeper in the early stage of training, and it is recommended to start with small-scale experiments for validation.
The research team has open-sourced Soft-MTP's code, paper, and HuggingFace model collection.
Soft-MTP can be implemented with one line of code, `h_ntp += h_mtp.detach() - h_mtp`, by modifying the residual connection of the hidden state.
The expression `h_mtp.detach() - h_mtp` is always numerically zero; the hidden state is unchanged during forward propagation, but during backpropagation the gradient flows through the MTP branch in negated form.
Soft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
Soft-MTP adds no forward or backward computation, introduces no new parameters, and does not change the model structure at inference time.
2 more timeline events
All Facts (12)
The change in gradient routing may affect training stability, especially when the loss surface is steeper in the early stage of training, and it is recommended to start with small-scale experiments for validation.
50%UnverifiedSoft-MTP can separate the KL self-distillation gradient with a one-line code change (h_ntp += h_mtp.detach() - h_mtp).
50%UnverifiedSoft-MTP does not require redesigning the model architecture or an additional training stage, and the existing MTP pretraining pipeline can be applied directly.
50%UnverifiedSoft-MTP improves MTP's performance across a variety of tasks.
50%UnverifiedThe benefits of Soft-MTP depend on the premise that the MTP head and the NTP head share the backbone network.
50%UnverifiedSoft-MTP can be implemented with one line of code, `h_ntp += h_mtp.detach() - h_mtp`, by modifying the residual connection of the hidden state.
50%UnverifiedThe expression `h_mtp.detach() - h_mtp` is always numerically zero; the hidden state is unchanged during forward propagation, but during backpropagation the gradient flows through the MTP branch in negated form.
50%UnverifiedSoft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
50%UnverifiedSoft-MTP adds no forward or backward computation, introduces no new parameters, and does not change the model structure at inference time.
50%UnverifiedSoft-MTP outperforms standard MTP on a variety of tasks, but the original material did not disclose specific benchmark data.
50%UnverifiedThe effectiveness of Soft-MTP depends on the training quality of the original MTP head; if the MTP head is underfit, the separable KL signal is limited.
50%UnverifiedThe research team has open-sourced Soft-MTP's code, paper, and HuggingFace model collection.
50%