Unverified50% confidenceOpinionExact time
The paper "Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation" points out that existing OPSD methods assume the teacher is fixed, and this assumption has fundamental limitations.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe core challenge of On-Policy Distillation (OPD) is how to extract effective supervisory signals from the teacher model; traditional methods rely on token probabilities or directly aligning hidden states, each with its own limitations: token probabilities may underestimate the teacher's internal knowledge, while directly aligning hidden states forces accommodation to the teacher's feature space, which is especially tricky in cross-architecture transfer.76% similarUnverifiedThe paper categorizes existing OPSD approaches into two types: one where the teacher is fixed throughout training and has access to the full reference; the other where the teacher is unchanged within each response but adaptively varies across responses in the degree of exposure to partial reference.74% similarUnverifiedOn-policy distillation requires the student to first generate its own prefix trajectory, after which the teacher provides guidance at the states the student actually reaches, so the state distributions are aligned, and it is usually more stable on long-horizon tasks where errors tend to accumulate.72% similarUnverifiedMTA-OPSD transforms the teacher selection problem in self-distillation into a token-level dynamic routing problem, where the teacher is no longer a static hyperparameter but a variable that changes with training state and token context.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1115268API
curl https://kongchang.com/api/v1/knowledge/claims/1115268MCP
get_claim(id=1115268)