AtomicRL
A reinforcement learning post-training framework for vision-language-action (VLA) models that exploits an atomic task structure to achieve fine-grained credit assignment and adaptive exploration, mitigating delayed credit assignment bias and uneven subtask mastery in long-horizon robotic manipulation.
Timeline (last 90 days)
AtomicRL is a reinforcement learning post-training framework that leverages atomic task structure to achieve reliable credit assignment and adaptive exploration, used to improve the performance of vision-language-action (VLA) models in long-horizon robotic manipulation.
AtomicRL's improvement does not replace PPO itself but introduces semantic-aware gating and bottleneck-adaptive sampling on top of it; the 3.6%–4.0% absolute gain reflects the benefit brought by structural credit-assignment improvements rather than a change of algorithm family.
The core of Semantic-Aware Policy Optimization (SAPO) is a semantic-aware gating mechanism that constrains TD residuals from failed atomic subtasks, preventing them from propagating across task boundaries.
Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives according to each atomic subtask's mastery level; poorly mastered subtasks receive more sampling opportunities and exploration rewards.
On the LIBERO-LONG benchmark, AtomicRL achieves absolute performance improvements of 3.6% and 4.0% over PPO on two VLA architectures, respectively.
On CALVIN ABC-D, AtomicRL improves full-task performance over PPO by up to 9.5%.
When combined with the Real2Sim2Real pipeline, AtomicRL improves real-world task performance by an average of 21.3%.
AtomicRL's average 21.3% real-world improvement depends on the effectiveness of the Real2Sim2Real transfer pipeline, whereas the improvements on simulation benchmarks mainly come from SAPO and BATE's improvements to credit assignment and exploration, so the two have different sources of contribution.
AtomicRL provides a path to continuously optimize VLA policies through online RL rather than relying on large amounts of expert demonstrations.
All Facts (9)
AtomicRL is a reinforcement learning post-training framework that leverages atomic task structure to achieve reliable credit assignment and adaptive exploration, used to improve the performance of vision-language-action (VLA) models in long-horizon robotic manipulation.
50%UnverifiedAtomicRL's improvement does not replace PPO itself but introduces semantic-aware gating and bottleneck-adaptive sampling on top of it; the 3.6%–4.0% absolute gain reflects the benefit brought by structural credit-assignment improvements rather than a change of algorithm family.
50%UnverifiedThe core of Semantic-Aware Policy Optimization (SAPO) is a semantic-aware gating mechanism that constrains TD residuals from failed atomic subtasks, preventing them from propagating across task boundaries.
50%UnverifiedBottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives according to each atomic subtask's mastery level; poorly mastered subtasks receive more sampling opportunities and exploration rewards.
50%UnverifiedOn the LIBERO-LONG benchmark, AtomicRL achieves absolute performance improvements of 3.6% and 4.0% over PPO on two VLA architectures, respectively.
50%UnverifiedOn CALVIN ABC-D, AtomicRL improves full-task performance over PPO by up to 9.5%.
50%UnverifiedWhen combined with the Real2Sim2Real pipeline, AtomicRL improves real-world task performance by an average of 21.3%.
50%UnverifiedAtomicRL's average 21.3% real-world improvement depends on the effectiveness of the Real2Sim2Real transfer pipeline, whereas the improvements on simulation benchmarks mainly come from SAPO and BATE's improvements to credit assignment and exploration, so the two have different sources of contribution.
50%UnverifiedAtomicRL provides a path to continuously optimize VLA policies through online RL rather than relying on large amounts of expert demonstrations.
50%