Sparse Reward

In sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.

Sparse rewards are one of the most challenging problems in reinforcement learning, where an agent only receives a reward upon completing a specific objective.

Common ways to mitigate sparse rewards include reward shaping, curiosity-driven exploration, hindsight experience replay (HER), etc.

Getting Over It is a typical sparse-reward environment, because only changes in height provide a reward signal and the intermediate process lacks guidance.

Curriculum learning is necessary for sparse-reward cooperative tasks; directly fading out the auxiliary rule-based policy is less robust than staged replacement.

The two agents cannot be trained from scratch simultaneously, because the tower arms have no learnable signal when the booster never appears, and the booster receives no catch reward when the tower arms never close.

The study suggests that practitioners must trade off computational cost against statistical accuracy when choosing the quantile resolution, especially when the reward distribution has vanishing endpoints, where an excessively high resolution may not bring the expected benefit.

In games like Super Mario, the agent often receives sparse rewards only upon clearing a level or dying, and intermediate steps lack clear correct-action labels, making the loss function difficult to define.

Neuroevolution does not rely on a differentiable loss function and is suitable for sparse-reward scenarios in reinforcement learning.

Temporal-difference-based methods struggle to attribute sparse final rewards to early actions, resulting in extremely weak learning signals.

The advantages of sparse rewards are low authoring cost and resistance to contamination by subjective bias; the disadvantage is that credit assignment is difficult, making it hard to determine which step in a long action sequence caused the failure.

Sparse rewards only provide signals when a task succeeds or fails, which leads to extremely low exploration efficiency in long-horizon, multi-stage tasks.

Overly granular intermediate rewards may guide the agent to pursue only local objectives while ignoring the globally optimal solution.

Long-horizon planning and sparse rewards are among the most difficult pain points for current reinforcement learning algorithms to overcome