Self-Distillation

Error messages should be written for the model to read and must be actionable, pointing out the actual mistake the model made rather than throwing raw parser errors, which can significantly improve the model's self-correction success rate.

仅用成功轨迹训练的模型往往存在分布外泛化的严重缺陷，遇到工具返回错误时便丧失纠偏能力

The teacher model in distillation does not have to be larger than the student model; it can be a domain expert model of comparable size.

In self-distillation, keeping two full model copies, teacher and student, resident on the GPU at the same time increases forward-pass latency and exacerbates ZeRO-3 communication overhead.

Self-distillation uses an exponential moving average (EMA) teacher to provide a stable, synchronous supervision signal.

Self-Distillation Policy Optimization (SDPO) provides dense token-level supervision signals to the student model through a feedback-conditioned "self-teacher".

Existing evaluation of self-distillation agent RL has structural problems: many works directly reuse baseline scores from different experimental conditions (different model scales, training steps, evaluation protocols) in published papers, and most papers compare only against their own pre-improvement versions, lacking direct comparison with other contemporaneous methods.

In on-policy distillation, the student model first generates trajectories itself, and then the teacher model provides supervision signals for these paths, thereby avoiding the distribution shift between offline data and student behavior.

RISED introduces a self-distillation mechanism that allows the model to extract supervision signals from its own high-quality trajectories, providing an additional learning source for samples that cannot be learned through group-relative rewards.

Self-Distillation eliminates the need for a separate teacher model, allowing the model to learn from its own historical checkpoints or high-quality outputs generated during current inference.

The UniEvo-VL project proposes enabling vision-language models to achieve self-improvement through self-distillation training, reducing reliance on external supervision signals.

The core difference between self-distillation and traditional knowledge distillation is that the teacher and student can be the same model, or outputs produced by the model itself at different training stages or from different perspectives.

The three main methods of self-distillation are Self-Distillation Fine-Tuning (SDFT), Self-Distillation Preference Optimization (SDPO), and on-policy self-distillation.