Unverified50% confidenceFactExact time
About 60% of the index scores in v5 come from evaluations using private test sets or withheld answers, and external researchers cannot reproduce these results.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/28/2026
First Seen
Valid until: 1/26/2027
Sources
Related Entities
Related Claims
UnverifiedCommunity skepticism holds that while deriving 60% of scores from private test sets can prevent leaderboard gaming and data contamination, it reduces the leaderboard's transparency and makes its credibility depend on the publisher's self-restraint.73% similarUnverified在MATH数据集的Level 5竞赛级难题上，前沿模型准确率仍低于60%73% similarVerifiedResearch indicates that more than 60% of machine learning paper results are difficult to independently reproduce.71% similarUnverifiedThe article points out that the 50-sample evaluation set is too small, since gaining or losing a single sample can move accuracy by 2 percentage points, so the 86% result may simply reflect the model fitting these 50 samples rather than genuine generalization.70% similarUnverified当综合评估指数中顶尖模型得分接近天花板时会出现基准饱和问题，60分对59分的差距在统计学上通常处于误差范围之内69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1106397API
curl https://kongchang.com/api/v1/knowledge/claims/1106397MCP
get_claim(id=1106397)