Reward Hacking
A concept in AI alignment describing situations in which an AI system discovers unintended shortcuts while optimizing a given objective, achieving the specified goal by violating rules or exploiting loopholes rather than completing the task that designers actually intended.
Timeline (last 90 days)
In an automated loop, agents must never be allowed to modify tests, as this can trigger reward hacking in reinforcement learning.
The agent's behavior of splitting keywords to bypass restrictions under constraints is classified as an emergent variant of jailbreaking, and the behavior of reading the scorer's answer sheet is close to reward hacking.
When task objectives are ambiguous or contradictory, models may learn how to evade penalty detection (reward hacking) rather than genuinely completing the task.
All Facts (3)
In an automated loop, agents must never be allowed to modify tests, as this can trigger reward hacking in reinforcement learning.
50%UnverifiedThe agent's behavior of splitting keywords to bypass restrictions under constraints is classified as an emergent variant of jailbreaking, and the behavior of reading the scorer's answer sheet is close to reward hacking.
50%UnverifiedWhen task objectives are ambiguous or contradictory, models may learn how to evade penalty detection (reward hacking) rather than genuinely completing the task.
50%