RLDS
A reinforcement learning method that improves credit assignment by decomposing trajectory rewards along subtask dimensions. Its core component, the Subtask Decomposition Advantage Estimator (SDAE), replaces the scalar GRPO advantage function to provide more precise token-level gradient signals.
Timeline (last 90 days)
For tasks with homogeneous skill composition and relatively dense feedback, the value of fine-grained reward decomposition is limited.
An arXiv paper proposes the RLDS (Reinforcement Learning with Decomposed Subtasks) method, whose core argument is that trajectory rewards should be decomposed along the subtask dimension before entering the policy update.
The core component of RLDS is the Subtask-Decomposed Advantage Estimation (SDAE), designed as a replacement for the scalar GRPO advantage function.
The authors evaluated RLDS on four agent benchmarks: FrozenLake, HotpotQA, ScienceWorld, and DeepResearch.
RLDS achieved an improvement of +11.5 points on ScienceWorld (95% confidence interval [+9.8, +13.3]) and +9.8 points on FrozenLake ([+7.0, +12.8]).
The improvements of RLDS on HotpotQA and DeepResearch fall within the noise range, consistent with the diagnostic prediction that these two tasks have limited recoverable information.
The gains from RLDS are amplified as subtask heterogeneity increases, with the most significant improvements on high-heterogeneity tasks.
RLDS reduces wall-clock time per step on ScienceWorld by 10.9% compared to scalar GRPO, because long trajectories amortize the fixed overhead of reflection and scoring.
RLDS（Reinforcement Learning Datasets）是Google DeepMind提出的基于TFRecord的标准化格式，用于存储强化学习和机器人学习的交互数据
All Facts (9)
For tasks with homogeneous skill composition and relatively dense feedback, the value of fine-grained reward decomposition is limited.
50%UnverifiedAn arXiv paper proposes the RLDS (Reinforcement Learning with Decomposed Subtasks) method, whose core argument is that trajectory rewards should be decomposed along the subtask dimension before entering the policy update.
50%UnverifiedThe core component of RLDS is the Subtask-Decomposed Advantage Estimation (SDAE), designed as a replacement for the scalar GRPO advantage function.
50%UnverifiedThe authors evaluated RLDS on four agent benchmarks: FrozenLake, HotpotQA, ScienceWorld, and DeepResearch.
50%UnverifiedRLDS achieved an improvement of +11.5 points on ScienceWorld (95% confidence interval [+9.8, +13.3]) and +9.8 points on FrozenLake ([+7.0, +12.8]).
50%UnverifiedThe improvements of RLDS on HotpotQA and DeepResearch fall within the noise range, consistent with the diagnostic prediction that these two tasks have limited recoverable information.
50%UnverifiedThe gains from RLDS are amplified as subtask heterogeneity increases, with the most significant improvements on high-heterogeneity tasks.
50%UnverifiedRLDS reduces wall-clock time per step on ScienceWorld by 10.9% compared to scalar GRPO, because long trajectories amortize the fixed overhead of reflection and scoring.
50%UnverifiedRLDS（Reinforcement Learning Datasets）是Google DeepMind提出的基于TFRecord的标准化格式，用于存储强化学习和机器人学习的交互数据
50%