Unverified60% confidenceFactExact time
Multiple independent studies in 2023 showed that some top models scored extremely high on standard mathematical reasoning tests but performed significantly worse on equivalent rephrased versions, indicating that the models learned test patterns rather than genuine reasoning ability.
2
Sources
60%
Confidence
Long-term
Relevance
7/15/2026
First Seen
Sources
Zig作者批评Anthropic：AI宣传与技术现实的落差
Related Claims
UnverifiedTraditional benchmarks have become ineffective for evaluating model reasoning ability because almost every model may have seen the standard answers in its training data.78% similarUnverifiedWhen researchers run different testing frameworks using the same model, scores often fluctuate noticeably.75% similarUnverified2022年Hoffman等人的Chinchilla论文重新校准了算力最优的参数-数据比例，指出此前许多大模型训练不充分75% similarUnverifiedThere is no stable mapping between a model's self-evaluation score and its actual accuracy; rephrase the same prompt and the score may drift substantially.74% similarUnverifiedIn an ablation experiment, after replacing the model's latent thinking segments with segments from unrelated questions, accuracy dropped by 45%–51%, indicating that latent thinking carries substantive reasoning.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/515034API
curl https://kongchang.com/api/v1/knowledge/claims/515034MCP
get_claim(id=515034)