Strong-to-Weak Distillation More Compute-Efficient Than GRPO? New Study Reveals Key Conditions

As reinforcement learning (RL) dominates large-model post-training, On-policy Distillation (OPD) offers a different path: letting the student learn from its own rollouts under per-token supervision from a frozen teacher. Can this dense signal reach target accuracy as efficiently as GRPO? A systematic empirical study offers a nuanced answer: the efficiency advantage depends heavily on the teacher source and task scenario.

Study Design: Cross-Dimensional Compute–Accuracy Measurement

The study compares vanilla reverse-KL OPD against Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher scale, domain, and optimization choices. The evaluation metric is not final accuracy alone but the ratio of held-out accuracy to training FLOPs — i.e., accuracy gain per unit of training compute. This measure better reflects decision-making under real training budgets.

The key variable is the teacher model: either an external post-trained teacher, or the student's own GRPO checkpoint. This distinction ultimately becomes central to understanding the efficiency difference between the two methods.

On-policy distillation (OPD) differs from traditional offline distillation: the student model updates on rollout trajectories it generates itself, under per-token supervision from a frozen teacher, so the training distribution stays aligned with the student's own policy. The vanilla reverse-KL OPD used here takes reverse KL divergence as the distillation loss — minimizing the reverse KL of the student distribution with respect to the teacher, which tends to make the student cover the teacher's high-probability modes rather than fit the mean. GRPO (Group Relative Policy Optimization) is a value-network-free policy gradient method: it samples a group of responses to the same prompt and uses the relative advantages of rewards within the group as the update signal — a typical on-policy RL post-training paradigm. Both rely on on-policy rollouts, but the density of supervision differs — OPD provides dense per-token teacher signals, while GRPO uses only sequence-level rewards. This difference is the basis for understanding the subsequent compute-efficiency comparison.

External Teacher: OPD's Compute Disadvantage

When the teacher comes from an external post-trained model, OPD generally underperforms GRPO under matched settings. This conclusion holds even when the teacher itself has been further trained with GRPO. More troublesome, OPD also frequently exhibits response-length inflation, meaning the model tends to generate longer outputs, further raising inference cost.

In other words, dense per-token supervision does not automatically translate into higher compute efficiency. The distribution gap between the external teacher and the student may force the student to expend more compute fitting the teacher signal, and that compute does not convert equivalently into accuracy.

Response-length inflation refers to the phenomenon in which the student's output length keeps growing during distillation. Because teacher models often prefer more detailed, longer reasoning chains, per-token reverse KL rewards the student for imitating this verbose style, raising average response length. In RL post-training, longer length directly increases FLOPs during rollout and inference latency, and this cost does not appear in the distillation loss — it is a hidden overhead. More critically, length inflation may accompany accuracy stagnation: the student learns the teacher's "form" without acquiring its "capability," further diluting the accuracy gain per unit of compute. This is also why evaluating OPD efficiency requires monitoring the response-length distribution rather than only final accuracy.

Self-Distillation Scenario: A 50× Compute Reduction Reversal

When the teacher is replaced by the student's own GRPO checkpoint, the picture flips. On math tasks, OPD can recover that checkpoint's accuracy with distillation FLOPs only 1/50 of what GRPO needs to reach the same accuracy. This is an order-of-magnitude gap, not a marginal improvement.

This finding is validated across multiple dimensions: the same high-efficiency pattern is consistently observed on larger students, code tasks, and a second model family. The team calls this setting "strong-to-weak" OPD: distilling the capability a checkpoint has already acquired through RL into a new student.

Strong-to-weak distillation refers to the setting in which a more capable model serves as teacher and a weaker or newly initialized student is the distillation target. What is special here is that the teacher is not an external model but the student's own checkpoint obtained after GRPO training — meaning teacher and student share initialization and data distribution, and the distribution gap is far smaller than in the external-teacher scenario. In this case, OPD's dense per-token signal essentially "compresses" the policy the RL checkpoint has already acquired back into a new student, rather than transferring capability across architecture or pretraining differences. Distribution proximity explains why equal distillation compute can recover checkpoint accuracy: the student needs no extra compute to bridge the distribution shift between teacher and itself, so dense supervision converts efficiently into accuracy.

Compute Ledger for Single-Domain vs. Cross-Domain

The efficiency advantage is not unconditional. Within a single domain, the GRPO checkpoint is already the target model, so distilling it again simply adds computation, and the total bill does not pay off. The real value appears in cross-domain scenarios: GRPO checkpoints from different domains each hold different capabilities, and no single checkpoint can serve all domains.

Multi-teacher OPD (DeepSeek^Ad-AI, 2026) consolidates domain-specific GRPO experts into one student. The distillation phase consumes only 5.2% (two domains) and 8.0% (four domains) of the experts' training FLOPs, yet the consolidated student matches each expert's accuracy. Counting expert training plus distillation together, the pipeline reaches joint-GRPO performance on two-domain mixtures with 42.0% fewer total FLOPs.

These numbers reveal a more pragmatic engineering logic: training domain experts separately and then consolidating them via low-cost distillation is more compute-efficient than directly training a multi-domain joint model. The OPD efficiency story is not about replacing RL, but about serving as a capability consolidation layer after RL.

Multi-teacher OPD refers to the setting in which multiple domain experts serve simultaneously as teachers to distill a single student. Unlike single-teacher distillation, the multi-teacher scenario must handle signal conflicts among teachers: different experts may give different probability distributions at the same token. Common approaches include weighted summation of each teacher's distillation loss, or routing samples to the corresponding teacher by domain. The two-domain and four-domain experiments here show that the distillation phase needs only 5.2% and 8.0% of expert training FLOPs for the consolidated student to match each expert's accuracy. The premise of this low cost is that experts' capabilities are complementary rather than overlapping — if experts' capabilities overlap heavily, the marginal benefit of multi-teacher consolidation decays quickly, and the total compute ledger loses its advantage.

Practical Implications

For training-pipeline designers, the study suggests clear decision boundaries:

• If the goal is single-domain capability, train directly with GRPO; extra distillation only adds total computation.
• If multi-domain capability is needed, domain-specific GRPO experts plus multi-teacher OPD consolidation beats joint training on the compute ledger.
• Distillation from external teachers requires careful evaluation of the hidden costs of response-length inflation.

OPD and GRPO are not substitutes but play different roles in the post-training pipeline. Understanding this division of labor is more practically valuable than arguing over which method is "better."

One-line Summary

OPD efficiency depends on the teacher source: external teachers lose to GRPO, while self-distillation and cross-domain consolidation can substantially save compute.

Paragraph Summary

This study systematically compares On-policy Distillation (OPD) and GRPO in efficiency as measured by the ratio of held-out accuracy to training FLOPs. Key findings: when the teacher is an external post-trained model, OPD generally underperforms GRPO and often suffers response-length inflation that raises hidden costs; when the teacher is the student's own GRPO checkpoint, OPD recovers equal accuracy on math tasks with about 1/50 of the distillation FLOPs, replicated on larger models, code tasks, and a second model family. The efficiency advantage has boundaries: single-domain distillation of one's own checkpoint is pure extra overhead, and the real value lies in cross-domain scenarios — multi-teacher OPD cuts expert consolidation cost to 5.2% (two domains) to 8.0% (four domains) of expert training FLOPs, with two-domain mixtures using 42.0% fewer total FLOPs than joint GRPO. OPD should be positioned as a capability consolidation layer after RL, not a replacement.