Unverified60% confidenceTradeoffExact time
The greatest risk of self-improvement comes from the reward signal itself: if the reward is improperly defined, the agent will optimize evaluation metrics rather than the true objective (the reinforcement learning version of Goodhart's Law).
2
Sources
60%
Confidence
Long-term
Relevance
8/7/2026
First Seen
Sources
Related Claims
UnverifiedWhether reward shaping can take effect depends on whether it changes the relative advantage among samples, not the absolute reward value.79% similarUnverifiedIn reinforcement learning, the reward signal is a proxy for the designer's intent, not the intent itself.78% similarUnverifiedIn domains with objective success or failure criteria, using subjective satisfaction as an optimization target is dangerous, and evaluation signals should be grounded in business ground truth78% similarUnverifiedIncentive distortion refers to when individual incentives and collective goals are misaligned, participants optimize the rewarded metrics rather than the genuinely valuable goals.76% similarUnverifiedThe core structural feature of prize mechanisms is risk transfer: the funder only pays upon success, while the cost of failure is borne by the participants75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/701165API
curl https://kongchang.com/api/v1/knowledge/claims/701165MCP
get_claim(id=701165)