Goodhart's Law
A classic principle in economics and AI: when a measure becomes a target, it ceases to be a good measure of the original goal. In AI alignment, it is commonly used to explain phenomena such as reward hacking and other forms of objective misspecification.
Core Facts
Benchmark overfitting is a manifestation of Goodhart's Law in AI evaluation: when a measurement becomes an optimization target, it loses its validity as a measure.
Reward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
Timeline (last 90 days)
Reinforcement learning is prone to reward hacking, where the model learns to game the reward model, which is academically referred to as Goodhart's Law.
Goodhart's Law dictates that once a specialized benchmark becomes an industry standard, model vendors will optimize specifically for it, causing it to lose its representativeness of real business needs.
In their 2018 paper, Manheim and Garrabrant subdivided Goodhart's Law into four types: regressive, extremal, causal, and adversarial
Reward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
In AI research, Goodhart's Law has been subdivided into four variants: regressional effect, extremal effect, causal misalignment, and adversarial gaming.
All Facts (7)
Benchmark overfitting is a manifestation of Goodhart's Law in AI evaluation: when a measurement becomes an optimization target, it loses its validity as a measure.
80%VerifiedReward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
65%UnverifiedThe theoretical root of reward hacking is Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.
60%UnverifiedReinforcement learning is prone to reward hacking, where the model learns to game the reward model, which is academically referred to as Goodhart's Law.
50%UnverifiedGoodhart's Law dictates that once a specialized benchmark becomes an industry standard, model vendors will optimize specifically for it, causing it to lose its representativeness of real business needs.
50%UnverifiedIn their 2018 paper, Manheim and Garrabrant subdivided Goodhart's Law into four types: regressive, extremal, causal, and adversarial
50%UnverifiedIn AI research, Goodhart's Law has been subdivided into four variants: regressional effect, extremal effect, causal misalignment, and adversarial gaming.
50%