Unverified50% confidenceFactExact time
In long-sequence ODP scenarios, when the student's generated prefix deviates from the teacher's own reasoning path, the quality of the teacher's feedback on subsequent tokens drops significantly, and parameter updates based on this feedback reinforce the deviation, forming a vicious cycle that leads to training instability.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedUnder off-policy sampling conditions, when the preference margin is aligned in direction with the utility gradient, DPO's update shifts from a corrective signal to a reinforcement-style update similar to reinforcement learning.74% similarUnverifiedFailure-driven PPO fine-tuning may be inferior to plain continuation training, with possible causes including distribution shift, reward-shaping side effects, and curriculum order sensitivity.74% similarVerifiedBehavioral cloning simplifies the problem to supervised learning, but suffers from distribution shift, where differences between the state distributions at training time and execution time lead to error accumulation.74% similarUnverifiedThe mismatch between P and Q introduces a drift term, causing model weights to be incorrectly updated even without a learning signal.72% similarUnverifiedValidation loss is computed under teacher-forcing mode, which differs from autoregressive generation at inference time, and therefore cannot expose failure modes such as error accumulation and repetition loops during generation71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1114713API
curl https://kongchang.com/api/v1/knowledge/claims/1114713MCP
get_claim(id=1114713)