ASTEROID
A training framework for partially observable reinforcement learning environments that uses asymmetric on-policy distillation to enable a student model to learn test-time exploration behavior under the guidance of a privileged expert.
Timeline (last 90 days)
The ASTEROID framework is proposed by the paper "Asymmetric On-Policy Distillation Learns In-Context Exploration" and is a student-teacher supervised learning scheme that enables a history-conditioned student model to learn test-time exploration through asymmetric on-policy distillation.
The ASTEROID training pipeline consists of two steps: the student model collects online interaction trajectories relying only on historical observations; the privileged expert, with access to the true hidden state, provides action labels based on the student's history context for distillation.
The paper evaluates ASTEROID in 11 different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments.
Under a fixed exploration budget, ASTEROID achieves about twice the return of baseline methods.
Compared with RL baselines, ASTEROID requires about 100 times fewer interactions.
ASTEROID surpasses frontier models in exploration capability.
After training in simulation, the ASTEROID policy successfully transferred to vision-limited real-world robotic manipulation tasks, improving the success rate by +40% compared with standard distillation methods.
The behavior of the ASTEROID student model is guided by expert labels toward a mode approximating Bayesian posterior sampling: when information is insufficient, it selects actions that can distinguish different hidden states, and when information is sufficient, it directly exploits.
The paper notes that the analogy between ASTEROID and posterior sampling may imply that the method inherits the theoretical guarantees of posterior sampling (such as logarithmic regret bounds), although the actual implementation does not explicitly maintain a belief distribution.
The ASTEROID method still requires a privileged expert to provide action labels, and the paper does not discuss the cost of expert annotation in detail.
1 more timeline events
All Facts (11)
ASTEROID's real-world transfer involves only vision-limited robotic manipulation, and broader scenarios such as autonomous driving and medical decision-making have not yet been validated.
50%UnverifiedThe ASTEROID training pipeline consists of two steps: the student model collects online interaction trajectories relying only on historical observations; the privileged expert, with access to the true hidden state, provides action labels based on the student's history context for distillation.
50%UnverifiedThe paper evaluates ASTEROID in 11 different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments.
50%UnverifiedUnder a fixed exploration budget, ASTEROID achieves about twice the return of baseline methods.
50%UnverifiedCompared with RL baselines, ASTEROID requires about 100 times fewer interactions.
50%UnverifiedASTEROID surpasses frontier models in exploration capability.
50%UnverifiedAfter training in simulation, the ASTEROID policy successfully transferred to vision-limited real-world robotic manipulation tasks, improving the success rate by +40% compared with standard distillation methods.
50%UnverifiedThe behavior of the ASTEROID student model is guided by expert labels toward a mode approximating Bayesian posterior sampling: when information is insufficient, it selects actions that can distinguish different hidden states, and when information is sufficient, it directly exploits.
50%UnverifiedThe ASTEROID framework is proposed by the paper "Asymmetric On-Policy Distillation Learns In-Context Exploration" and is a student-teacher supervised learning scheme that enables a history-conditioned student model to learn test-time exploration through asymmetric on-policy distillation.
50%UnverifiedThe ASTEROID method still requires a privileged expert to provide action labels, and the paper does not discuss the cost of expert annotation in detail.
50%UnverifiedThe paper notes that the analogy between ASTEROID and posterior sampling may imply that the method inherits the theoretical guarantees of posterior sampling (such as logarithmic regret bounds), although the actual implementation does not explicitly maintain a belief distribution.
50%