Improving On-Policy Self-Distillation for Thinking Models with Activation Steering

Introduction

On-policy Self-Distillation (OPSD) is an effective method for training thinking models: the student model receives token-level supervision on its own reasoning trajectories, and the supervision signal comes from a teacher model initialized with a reference condition—this teacher model shares the same origin as the student model. However, the presence of this reference condition may induce the teacher model to exhibit behaviors that are detrimental to independent reasoning, which are then passed on to the student. A recent study proposed Steered-Teacher OPSD (ST-OPSD), which uses activation steering to simultaneously mitigate two key failure modes: reference hallucination and reconsideration suppression.

On-policy Self-Distillation (OPSD) differs from traditional offline knowledge distillation in that it emphasizes the student model learning from its own generated trajectories (on-policy) rather than trajectories from a fixed dataset. In the training of thinking models, OPSD typically sets up a teacher model that shares the same origin as the student, and this teacher model has access to additional reference information (such as standard answers or solution hints) and provides token-level soft labels along the reasoning paths that the student itself explores. This same-origin teacher avoids distribution mismatch caused by differences in model capacity, while leveraging reference information to improve supervision quality—but the presence of the reference condition also introduces the risk of the teacher's behavior deviating, which is precisely the starting point of this article's discussion.

Problem: Two Failure Modes Introduced by the Reference Condition

The core of OPSD is that the teacher model has access to reference information, thereby providing the student with more accurate token-level distributions. But the problem also stems from this: the teacher model may over-rely on the reference, exhibiting two harmful behaviors.

Reference Hallucination

The student model cites an unavailable reference to justify its reasoning steps during inference, i.e., "reference hallucination." This undermines the independence of reasoning; the student may fabricate reference content to support its own conclusions rather than truly deriving them based on the problem itself.

Reference hallucination is somewhat different from what is commonly called "factual hallucination." Factual hallucination refers to the model generating content that is inconsistent with facts, whereas reference hallucination specifically refers to the model fabricating or incorrectly citing an external reference source during reasoning (for example, claiming "according to the reference answer..." or "the problem hint points out..."), even when that reference is unavailable or does not exist at inference time. In OPSD, because the teacher model habitually relies on the reference condition during training, it may pass this reliance pattern on to the student, causing the student to also fabricate reference grounds during independent reasoning, thereby weakening logical derivation based on the problem itself. Detecting reference hallucination typically requires identifying explicit or implicit invocations of references in reasoning trajectories and determining whether they are verifiable.

Reconsideration Suppression

At uncertain steps, the teacher model may discourage verification and correction behaviors, i.e., "reconsideration suppression." Thinking models should be able to backtrack, verify, and correct their reasoning, but the teacher's supervision signal may suppress this ability, causing the student model to choose to keep moving forward when it should reconsider.

Both failure modes weaken the student model's independent reasoning ability, running counter to the original intent of OPSD.

Method: How ST-OPSD Steers the Teacher Model

The core idea of ST-OPSD is: steer only the teacher model, not the student model. Specifically, it extracts two direction vectors from the model-generated reasoning trajectories and applies these directions to the teacher model's hidden states when computing the distillation objective, thereby suppressing reference-invoking behavior and encouraging reconsideration.

Activation Steering and Direction Extraction

The research team extracted two directions from the model-generated reasoning trajectories: one for suppressing reference-invoking behavior and another for encouraging reconsideration. These directions are applied to the teacher model's hidden states, while the student model itself is not steered. The student learns from the steered teacher distribution, thereby indirectly obtaining more reasonable supervision signals.

Activation steering is a technique that controls model behavior without fine-tuning model parameters. The basic approach is to add a direction vector (steering vector) to the hidden state vector of a specific layer during the model's forward pass, thereby enhancing or suppressing a certain semantic attribute. Direction vectors are usually extracted by comparing the means of hidden states under different behaviors (e.g., the difference between positive and negative examples), and can also be obtained with the help of linear probes or principal component analysis. Since it only intervenes in activation values at inference time, activation steering is lightweight, interpretable, and reversible, making it suitable for temporarily correcting the teacher model's behavior during distillation without changing its weights.

Marker-family Calibration

ST-OPSD also introduces marker-family calibration, used to enhance the encouraging effect on reconsideration. The specific implementation details are not elaborated in the original text, but the method aims to ensure that the steering signal remains consistent and effective across different marker families.

A key design is: the student model is not steered during the learning process. This means the student still generates trajectories according to its own policy, while the supervision signals provided by the teacher model become more reasonable after steering. This asymmetric design avoids intervening in the student model at inference time, preserving the independence of reasoning.

Experimental Results

The study conducted experiments on two mathematical reasoning benchmarks, AIME and HMMT. The results show that ST-OPSD improves accuracy compared with the original OPSD and other intervention methods, while restoring reconsideration and reference hallucination rates to the base model level.

This means that ST-OPSD not only improves task performance but also fixes the behavioral deviations introduced by OPSD, making the student model's reasoning behavior closer to that of the undistilled base model.

AIME (American Invitational Mathematics Examination) and HMMT (Harvard-MIT Mathematics Tournament) are high-difficulty competition benchmarks commonly used in mathematical reasoning evaluation. AIME contains 15 problems with integer answers, emphasizing the integrated use of algebra, geometry, number theory, and combinatorics; HMMT includes various formats such as team rounds and individual rounds, with more diverse problem difficulty and style. Both are often used to evaluate the ability of thinking models in complex multi-step reasoning, because they require the model not only to give the final answer but also to maintain the coherence and verifiability of the reasoning chain, and are therefore also suitable for observing behavioral metrics such as reconsideration and reference hallucination.

Analysis and Implications

The value of ST-OPSD lies in revealing an important problem: the teacher signal during distillation may be "contaminated" by the reference condition, and simple token-level supervision is insufficient to solve this problem. By directly intervening in the teacher's hidden states through activation steering, ST-OPSD provides a lightweight solution that does not require modifying the student model's architecture or inference process.

This study also reminds us that when training thinking models, attention should be paid to the behavioral quality of the teacher model, not just the final accuracy. Reference hallucination and reconsideration suppression are two subtle but far-reaching failure modes that may quietly reduce the model's reasoning reliability without affecting the training loss.

Future work can further explore: how to automatically discover and extract directions for more harmful behaviors? How to generalize this method to more types of reference-conditioned tasks? And can the specific mechanism of marker-family calibration be applied more broadly?

Conclusion

ST-OPSD improves on-policy self-distillation through activation steering, specifically addressing the two major problems of reference hallucination and reconsideration suppression. Experiments show that this method improves accuracy while restoring the model's independent reasoning behavior. This work provides a new perspective and a practical tool for the distillation training of thinking models.

One-line Summary

ST-OPSD uses activation steering to correct the teacher model, addressing reference hallucination and reconsideration suppression in on-policy self-distillation.

Paragraph Summary

This article introduces Steered-Teacher OPSD (ST-OPSD), a method for improving on-policy self-distillation (OPSD) training of thinking models. In OPSD, because the teacher model has access to the reference condition, it may produce two failure modes—reference hallucination and reconsideration suppression—and pass the harmful behaviors on to the student. The core of ST-OPSD is to steer only the teacher model: extract direction vectors from reasoning trajectories to suppress reference invocation and encourage reconsideration, apply them to the teacher's hidden states when computing the distillation objective, and introduce marker-family calibration to enhance the effect. The student model is not steered; it still generates trajectories according to its own policy and learns from the corrected teacher distribution. On the AIME and HMMT mathematical reasoning benchmarks, ST-OPSD improves accuracy compared with the original OPSD and restores reconsideration and reference hallucination rates to the base model level, improving task performance while fixing behavioral deviations.