Unverified50% confidenceFactExact time
The leaderboards themselves can also be specifically optimized for, the so-called "leaderboard gaming," making score improvements not equivalent to actual capability improvements.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen evaluating a metric, ask: if a number is optimized to an extreme value, does the real goal improve in tandem? If the answer is no, then it is at best a vanity metric, and using it to drive an agent will inevitably lead to goal drift.77% similarUnverifiedThe gains from deductive and abductive prompting strategies are uneven and highly dependent on the specific dimension.76% similarUnverifiedReports have indicated that some vendors specifically optimize their outputs for the Arena style, which is a manifestation of Goodhart's Law — when a metric becomes a target, it ceases to be a good metric.75% similarUnverifiedOptimizing for benchmarks carries risks, as high scores may only reflect a model's adaptation to specific question types rather than a genuine improvement in general capabilities.75% similarUnverifiedThe effectiveness of automatic hill-climbing metric optimization is highly dependent on the quality of the chosen evaluation metric, may fall into the limitations of the metric itself, and requires researchers to intervene with judgment at critical points.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1071736API
curl https://kongchang.com/api/v1/knowledge/claims/1071736MCP
get_claim(id=1071736)