On-policy distillation
A knowledge distillation paradigm in which the student model receives token-level supervision from the teacher on trajectories it samples in real time under its current parameters. In contrast to offline distillation, which learns from fixed sequences, it aims to align the training and inference distributions.
Timeline (last 90 days)
On-policy distillation (OPD) is a variant of knowledge distillation in which training data is generated in real time by the student model under its current parameters, and the teacher model provides per-token probability distributions over the self-generated responses as supervision signals.
In long-sequence ODP scenarios, when the student's generated prefix deviates from the teacher's own reasoning path, the quality of the teacher's feedback on subsequent tokens drops significantly, and parameter updates based on this feedback reinforce the deviation, forming a vicious cycle that leads to training instability.
The training data for on-policy distillation is generated by real-time sampling of the student model under its current parameters, rather than being a pre-prepared fixed corpus, which contrasts with offline distillation.
The paper finds that when the student's trajectory moves in the wrong direction, the teacher does not correct the error but instead continues writing along the student's error, that is, the teacher has the ability to recognize errors but lacks the ability to escape from them.
The authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
All Facts (5)
On-policy distillation (OPD) is a variant of knowledge distillation in which training data is generated in real time by the student model under its current parameters, and the teacher model provides per-token probability distributions over the self-generated responses as supervision signals.
50%UnverifiedIn long-sequence ODP scenarios, when the student's generated prefix deviates from the teacher's own reasoning path, the quality of the teacher's feedback on subsequent tokens drops significantly, and parameter updates based on this feedback reinforce the deviation, forming a vicious cycle that leads to training instability.
50%UnverifiedThe training data for on-policy distillation is generated by real-time sampling of the student model under its current parameters, rather than being a pre-prepared fixed corpus, which contrasts with offline distillation.
50%UnverifiedThe paper finds that when the student's trajectory moves in the wrong direction, the teacher does not correct the error but instead continues writing along the student's error, that is, the teacher has the ability to recognize errors but lacks the ability to escape from them.
50%UnverifiedThe authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
50%