FOCUS
A reinforcement learning method for credit assignment in long-horizon language model agents. Through selective branching and an active learning strategy, it trains a critic network efficiently within a sampling budget and combines TD(λ) advantage estimation to address credit assignment under sparse rewards.
Timeline (last 90 days)
The FOCUS method uses selective branching and active learning to alleviate two problems in credit assignment for long-horizon agents under sparse outcome rewards: inaccurate critic value estimates and the high cost of continuation sampling.
Under the condition of matched per-step generation budgets, FOCUS at the Qwen3.5-9B scale improves by 7.2 percentage points over the strongest baseline (the higher of GRPO/VinePPO) on the BrowseComp-Plus benchmark, reaching 55.4 versus 48.2.
FOCUS's selective branching has a rollout controller actively select the states expected to most reduce critic estimation error for continuation sampling, rather than sampling uniformly across all states.
FOCUS uses TD(λ) advantages to score segments between consecutive states, performing segment-level credit assignment combined with refined value estimates.
FOCUS decouples context management from rollout allocation, so that segment boundaries produced by context compression can be directly used for value estimation and credit assignment.
FOCUS's experiments are concentrated on 9B-scale models and specific benchmarks, and the method's generalization to larger models and broader tasks remains to be verified.
FOCUS conducts experiments on three types of benchmarks — legal reasoning, deep search, and code localization — comparing against GRPO and VinePPO, and outperforms both.
All Facts (7)
The FOCUS method uses selective branching and active learning to alleviate two problems in credit assignment for long-horizon agents under sparse outcome rewards: inaccurate critic value estimates and the high cost of continuation sampling.
50%UnverifiedUnder the condition of matched per-step generation budgets, FOCUS at the Qwen3.5-9B scale improves by 7.2 percentage points over the strongest baseline (the higher of GRPO/VinePPO) on the BrowseComp-Plus benchmark, reaching 55.4 versus 48.2.
50%UnverifiedFOCUS's selective branching has a rollout controller actively select the states expected to most reduce critic estimation error for continuation sampling, rather than sampling uniformly across all states.
50%UnverifiedFOCUS uses TD(λ) advantages to score segments between consecutive states, performing segment-level credit assignment combined with refined value estimates.
50%UnverifiedFOCUS decouples context management from rollout allocation, so that segment boundaries produced by context compression can be directly used for value estimation and credit assignment.
50%UnverifiedFOCUS's experiments are concentrated on 9B-scale models and specific benchmarks, and the method's generalization to larger models and broader tasks remains to be verified.
50%UnverifiedFOCUS conducts experiments on three types of benchmarks — legal reasoning, deep search, and code localization — comparing against GRPO and VinePPO, and outperforms both.
50%