Unverified50% confidenceFactExact time
JOLT's design eliminates the overhead of maintaining two sets of parameters and lets the teacher signal evolve in sync with the student policy, avoiding the teacher lagging behind the student's current capability level due to being frozen.
1
Sources
50%
Confidence
Long-term
Relevance
10/22/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe paper observes that supervision signals from a fixed teacher become increasingly less worthwhile as the student model's capabilities improve67% similarUnverifiedExisting few-step distillation methods fall roughly into two categories: one relies on an additional score model to estimate the teacher distribution, and the other introduces adversarial losses to bring the student's output distribution closer to the teacher's.65% similarUnverifiedA distilled model can retain 70–90% of the teacher model's capabilities.65% similarUnverifiedOn-policy distillation requires the student to first generate its own prefix trajectory, after which the teacher provides guidance at the states the student actually reaches, so the state distributions are aligned, and it is usually more stable on long-horizon tasks where errors tend to accumulate.64% similarUnverifiedIn self-distillation, if the teacher is an EMA or synchronous variant of the student policy, each update of the student changes the teacher, causing the target signal to drift and turning training into chasing a moving target.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1084762API
curl https://kongchang.com/api/v1/knowledge/claims/1084762MCP
get_claim(id=1084762)