CoRD: How Coupled Sampling Fills the Blind Spot of On-Policy Distillation

Title

CoRD: How Coupled Sampling Fills the Blind Spot of On-Policy Distillation

Meta Description

CoRD proposes coupled rollout distillation, using the maximal coupling from speculative decoding to sample from an interpolation of student and teacher distributions, filling the blind spot of states the student never reaches in on-policy distillation. Across six student models of Qwen3 and Qwen3.5, both pass@1 and pass@8 outperform on-policy distillation, and the 4B student closes the gap within 16 steps that on-policy distillation fails to reach in 80 steps.

Article Content

On-policy distillation lets a student learn on its own trajectories, which introduces a fundamental blind spot: states the teacher can reach but the student rarely or never reaches receive almost no supervision signal. The paper "CoRD: Coupled Rollouts Fill the Blind Spot of On-Policy Distillation" proposes coupled rollout distillation (CoRD), which attempts to actively push the training-state distribution toward the teacher side without abandoning the on-policy loss.

Where the blind spot of on-policy distillation comes from

The core advantage of on-policy distillation is that the training distribution matches the distribution at inference time—the student only receives corrective teacher signals on its own generated token sequences. The problem arises precisely here: states the student cannot reach give the teacher no opportunity to teach. The teacher may have a clear preference for these states, but because the student's sampling probability is extremely low, these preferences cannot be converted into gradients. As training progresses, the student may become increasingly stable in regions it is familiar with while consistently avoiding the "teacher paths" that would truly make a difference.

The CoRD authors call this the blind spot of on-policy distillation. To fill it, the most direct approach is to have the student sample in regions the teacher prefers, but switching entirely to the teacher distribution would undermine the essence of on-policy distillation: once the training states deviate from the student's own distribution, the learned corrections may not transfer back to the student's own trajectories. The design goal of CoRD thus becomes clear: keep the on-policy loss and only shift the distribution of training states.

On-policy distillation is a training paradigm within knowledge distillation: the student model receives supervision signals from the teacher model on token sequences generated by its own policy, rather than learning on fixed human annotations or offline data generated by the teacher. This contrasts with classical offline distillation—in offline distillation, the distribution the student sees during training is determined by the teacher or the dataset, but at inference it faces its own generated distribution, creating a train–inference distribution shift. On-policy distillation eliminates this shift by letting the student "learn as it goes," and its loss is typically defined as the divergence (such as KL divergence) between the teacher and student distributions along the student's sampled trajectories. The cost is that the training states are entirely determined by the student's current policy, so regions with extremely low student probability mass are almost never sampled and thus receive no gradients.

State interpolation via maximal coupling from speculative decoding

CoRD samples trajectories from a geometric interpolation of the student and teacher distributions, and the interpolation is implemented using the maximal coupling from speculative decoding. Specifically, the coupling mechanism retains the student's own token at token positions where the two distributions agree, and at positions where they disagree, replaces a token from the portion of the target distribution that the student underestimates.

The elegance of this design lies in the intervention frequency. Maximal coupling ensures that, within the limits allowed by an exact sampler, interventions occur as rarely as possible, so the student's trajectory is not broadly overwritten by teacher signals. As the student gradually learns the tokens the teacher prefers, the two distributions converge, replacements naturally decrease, and the intervention strength automatically anneals with the learning process. In other words, CoRD does not forcibly pull the student onto the teacher distribution; it gives a gentle nudge only where the student "misses" something.

Maximal coupling is a way in probability theory to construct a joint distribution of two distributions: given two discrete distributions P and Q, the maximal coupling maximizes the probability that the two random variables are equal, which equals the sum of their overlapping probability mass. On language model token distributions, this means making the two sampling results agree as much as possible at positions where the probability mass of P and Q overlaps, and allocating according to the remaining mass only at points of disagreement. Speculative decoding exploits exactly this property: a small model quickly generates draft tokens, which a large model then verifies in parallel, and the accepted tokens correspond precisely to the parts where the two distributions agree. CoRD borrows the same mechanism, turning "accept/reject" into "keep the student token / replace a token from the disagreement region preferred by the teacher," thereby minimizing the number of replacements in theory and making the sampling distribution a controllable interpolation between the student and teacher distributions.

Empirical results on six student models

The paper validates CoRD on two model families and six base students, covering Qwen3 and Qwen3.5 with parameter counts from 0.6B to 4B. On five public math benchmarks, with the same number of training steps and the same amount of data, CoRD improves average pass@1 by 0.9 to 2.0 percentage points and pass@8 by 1.2 to 3.4 percentage points compared with on-policy distillation.

It is worth noting that the gain in pass@k is larger than that in pass@1. This indicates that CoRD does more than fine-tune the quality of a single sample—it enables the student to cover more correct paths that were previously overlooked. This is consistent with the method's motivation: what gets filled in are precisely the regions that the student's own distribution covers insufficiently.

The paper also designs an experiment that freezes the teacher-reachable states. Under this setting, the 4B student trained with CoRD achieves within 16 steps a gap reduction that on-policy distillation still fails to reach after 80 steps. The difference in step efficiency is quite significant, and it directly shows that moving the training-state distribution toward the teacher is more effective at reaching the blind spot than simply increasing the number of on-policy training steps.

pass@k is a commonly used evaluation metric in code and mathematical reasoning: for the same problem, sample k times independently, and count it as a pass if any one of the answers is correct, usually reporting the expected value over multiple experiments. pass@1 measures the accuracy of a single sample and reflects the model's average reliability; pass@k (k>1) measures the model's ability to find a correct solution within k attempts, and is closer to "coverage" than to "precision." When the gain in pass@k is significantly larger than in pass@1, it usually means the model generates more diverse correct reasoning paths, rather than merely raising the probability of correct answers that already appeared frequently. This distinction is crucial for judging whether a distillation method truly expands the model's problem-solving ability.

Ablation on replacement rules: why maximal coupling

A natural question is: if simply replacing some tokens during training can fill the blind spot, why insist on maximal coupling? The paper's ablation provides the answer. The authors tested replacement rules that intervene in other directions at a frequency no lower than CoRD's, and all performed worse than CoRD.

This shows that the effect does not come from "intervening more" but from "intervening accurately." Maximal coupling restricts replacements to positions where the two distributions genuinely disagree, and prioritizes replacing the target portion the student underestimates, so that each intervention carries the information the teacher truly intends to convey. More frequent but directionally arbitrary replacements may instead introduce noise and dilute the corrective ability of the on-policy loss.

Implications for distillation practice

The value of CoRD lies in redrawing the boundary between "preserving the student distribution" and "leveraging the teacher distribution" in on-policy distillation. Previous methods often made an either-or choice between these two ends: either strictly on-policy and accept the blind spot, or switch to offline or teacher forcing and sacrifice distribution matching. CoRD offers a third path: use coupled sampling to make the minimal necessary distribution shift, with the amount of intervention determined by the coupling structure of the two distributions rather than by manually set hyperparameters.

From an engineering perspective, the method reuses the existing machinery of speculative decoding, the replacement rule has theoretical guarantees from the exact sampler, and the intervention strength automatically decays with training—all of which favor practical deployment. The model scales covered by the paper, from 0.6B to 4B, also preliminarily demonstrate applicability across scales and model families. Worth watching going forward is whether this idea can transfer to reasoning tasks beyond mathematics, and how the intervention frequency and training stability of coupled sampling change when the capability gap between teacher and student is larger.

One-line Summary

CoRD uses maximal coupling sampling to push the training states of on-policy distillation toward the teacher, filling the supervision blind spot the student cannot reach.

Paragraph Summary

The paper addresses the fundamental blind spot of on-policy distillation—states the teacher can reach but the student rarely reaches lack supervision signals—by proposing coupled rollout distillation, CoRD. The method keeps the on-policy loss and uses only the maximal coupling from speculative decoding to sample from a geometric interpolation of the student and teacher distributions: it keeps the student token where the distributions agree and replaces a token from the target portion the student underestimates where they disagree, minimizing the number of interventions and automatically annealing as the student improves. Across six student models of Qwen3 and Qwen3.5 (0.6B to 4B), the average pass@1 on five math benchmarks improves by 0.9 to 2.0 points and pass@8 by 1.2 to 3.4 points; in the experiment freezing teacher-reachable states, the 4B student achieves within 16 steps a gap reduction that on-policy distillation still fails to reach after 80 steps. The ablation shows the effect comes from the accuracy of the interventions rather than their frequency.