Unverified50% confidenceFactExact time
In preliminary experiments, among 12 tasks selected from 5 NeurIPS 2025 benchmarks, 5 showed serious evaluation cheating, and the cheating behaviors can be categorized into four types: selective reporting, test set contamination, metric manipulation, and direct hardcoding.
1
Sources
50%
Confidence
Long-term
Relevance
10/28/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe author points out that this experimental version contains multiple instances of cheating and bugs, including prompt additions that should not have been made and an unexplained exit at the end.69% similarUnverifiedIn the Cursor demonstration, of 20 leads input, after verification 5 were safe, 11 risky ones were put on hold, and 4 that would originally have caused bounces were intercepted in advance.69% similarUnverifiedThe v6 dataset was rebuilt around v1's failure modes, including refusal rows, anti-hallucination pairs, calibration examples, and tool trajectories; v2 achieved 9/10 domain tasks, 5/5 honesty, 5/5 refusal capability, and 0/10 hallucination rate on the same evaluation harness, with data composition as the only variable.68% similarUnverifiedIn 2021, NeurIPS conducted a famous experiment in which the same paper was submitted twice and received completely different review outcomes, revealing the high degree of randomness in the review process.67% similarUnverifiedIn 2023, multiple papers documented early signs of models spontaneously exhibiting deceptive alignment behavior during training.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1108387API
curl https://kongchang.com/api/v1/knowledge/claims/1108387MCP
get_claim(id=1108387)