On-Policy Distillation

Model distillation refers to the process of querying a target model extensively, collecting its input-output pairs, and then training a student model with comparable capabilities.

Online policy distillation (OPD) has the student model generate its own training responses, which are then supervised by the teacher model.

In the supervised fine-tuning stage, Accordion integrates three domain experts into a single model through parameter merging and budget-matched multi-teacher on-policy distillation.

The training data for on-policy distillation is generated by the interaction of the student's current policy with the environment, and the state distribution changes as the student policy is updated, in line with the DAgger idea, which can mitigate distribution shift.

The ASTEROID training pipeline consists of two steps: the student model collects online interaction trajectories relying only on historical observations; the privileged expert, with access to the true hidden state, provides action labels based on the student's history context for distillation.

OPD does not require task success/failure reward signals and relies only on the teacher's token-by-token supervision, so it is a reward-free method.

The researchers found that even starting from the states where the student deviates the most, the teacher's own takeover can still complete the task, whereas the student, after training on the teacher's per-token scores, often runs out of turns.

The core challenge of On-Policy Distillation (OPD) is how to extract effective supervisory signals from the teacher model; traditional methods rely on token probabilities or directly aligning hidden states, each with its own limitations: token probabilities may underestimate the teacher's internal knowledge, while directly aligning hidden states forces accommodation to the teacher's feature space, which is especially tricky in cross-architecture transfer.

Token-level on-policy distillation aligns the student model's hidden states with teacher signals and adds a gating mechanism for credit assignment, yielding improvements on math and tool-use benchmarks.

The key mechanism of RM-OPD is applying token-level dense supervision on critiques generated by the student itself, making the supervision distribution consistent with the student's current policy distribution and alleviating the training-inference distribution mismatch problem.

The arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage process: first train multiple domain-specific reward model teachers, then integrate them into one generative student model through routed on-policy distillation, and this unified student model is said to surpass all of its distilled expert teachers.

On-policy distillation (OPD) has the student model generate its own trajectories, and then the teacher model provides supervision signals on those trajectories.

Version 2.0 switched to online policy distillation, where the model generates its own responses and a teacher model scores each token; two teachers were set up — the original model plus hidden human instructions for chat, and a pure base model for instruction following, tool calling, and coding.