Reward Shaping

奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径

Reward shaping refers to adding extra, denser guiding reward signals on top of the original sparse reward to help agents learn faster and more correctly.

In sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.

Sparse rewards are one of the most challenging problems in reinforcement learning, where an agent only receives a reward upon completing a specific objective.

约束驱动法的提示词策略只设定边界条件而不预设实现路径，与强化学习中的'奖励塑形'（reward shaping）理念一致

Common ways to mitigate sparse rewards include reward shaping, curiosity-driven exploration, hindsight experience replay (HER), etc.

The 1999 work by Ng et al. pointed out that inappropriate reward shaping can change the optimal policy, causing the agent to pursue auxiliary rewards instead of the true objective (reward hacking).

Means of mitigating reward hacking include reward shaping, multi-objective constraints, and introducing human judgment in reinforcement learning from human feedback (RLHF), but completely eliminating reward hacking remains an open problem.

Artificially constructing dense signals in a sparse reward space falls under the category of reward shaping in RL engineering.

Common approaches to alleviating the state sparsity of PDF environments include introducing a retrieval module to narrow the candidate range, using curriculum learning to transition from short documents to long documents, and designing intermediate rewards.

Reward shaping needs to be designed carefully to prevent the agent from exploiting reward loopholes and producing unintended behavior.

Training starts with sparse rewards, and then iterates through reward shaping via an event frequency table; in the event frequency table, movement penalties account for 48.3% of all events, exploring tiles accounts for 21.7%, and attacking monsters accounts for 14.5%.

Traditional approaches include introducing intermediate rewards, reward shaping, or hindsight relabeling (such as HER), but these methods either rely on manual design or are difficult to adapt to the highly heterogeneous scenarios in MLE where code execution feedback is complex.

Reward shaping refers to designing additional auxiliary reward terms beyond the sparse original reward to guide the policy, which can accelerate convergence, but if the auxiliary rewards are inconsistent with the final objective, the policy may game the score without truly improving the win rate; in practice a smaller weight is often multiplied in and decayed in the later stages of training.

Embedding the social marginal cost into each agent's individual reward is a reward shaping strategy that requires no modification to the underlying MARL algorithm architecture.

If the reward shaping signal is too weak, the AI will take shortcuts; if it is too strong or poorly designed, it may introduce new exploits or constrain the exploration space.

With the help of reward shaping and league play, the author ultimately trained agents to exhibit relatively interesting combat behavior.

Overly sparse rewards in Pokémon-type games make it difficult for agents to learn, so researchers typically introduce reward shaping strategies such as incentives for exploring new areas, rewards for catching Pokémon, and feedback for winning battles

Experiments found that the task_score under the shaped reward configuration was only 0.125, compared to 0.417 for the naive reward configuration, with an effect size of d=4.47.

The reward shaping method was first systematized by Andrew Ng and others in the 1999 paper "Policy Invariance Under Reward Transformations".