Unverified50% confidenceFactExact time
MTA-OPSD transforms the teacher selection problem in self-distillation into a token-level dynamic routing problem, where the teacher is no longer a static hyperparameter but a variable that changes with training state and token context.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe core design of MTA-OPSD is to construct a multi-teacher pool of candidate teachers and let each token adaptively select its preferred teacher during training.73% similarUnverifiedThe paper categorizes existing OPSD approaches into two types: one where the teacher is fixed throughout training and has access to the full reference; the other where the teacher is unchanged within each response but adaptively varies across responses in the degree of exposure to partial reference.72% similarUnverifiedOffline distillation trains the student using a static dataset pre-generated by the teacher model, which leads to the problem of covariate shift.68% similarUnverifiedThe paper "Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation" points out that existing OPSD methods assume the teacher is fixed, and this assumption has fundamental limitations.68% similarUnverifiedOn-policy distillation requires the student to first generate its own prefix trajectory, after which the teacher provides guidance at the states the student actually reaches, so the state distributions are aligned, and it is usually more stable on long-horizon tasks where errors tend to accumulate.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1115276API
curl https://kongchang.com/api/v1/knowledge/claims/1115276MCP
get_claim(id=1115276)