Unverified50% confidenceBenchmarkExact time
Of the 131 tasks, only 28 are numerical computation type, yet half of the twelve tasks on which all four models failed belong to numerical computation.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/20/2026
First Seen
Valid until: 1/18/2027
Sources
Related Entities
Related Claims
UnverifiedThe current dataset contains 770 recorded attempts across 29 model/configuration entries, but the unified test suite includes only 16 distinct tasks.76% similarUnverifiedEach model answered 112 questions (7 flaw types × 2 versions × 4 repetitions × 2 conditions), with half flawed and half sound, and a model that mechanically answered 'none' would receive a baseline score of 0.50.73% similarUnverifiedIn memory recall tests, the 12B model answered 82 out of 100 questions correctly and the 31B model answered 98 out of 100, with neither model hallucinating answers.72% similarUnverifiedIn a preregistered study, confidence intervals relying solely on LLM judges failed to cover the full human-annotation ground truth in 18 of 21 task-and-judge combinations.70% similarUnverifiedThe numbers task was the hardest for 6 of the 8 models, with scores on large numbers (10¹⁵ to 10³⁹) ranging from only 0.06 to 0.53.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1073467API
curl https://kongchang.com/api/v1/knowledge/claims/1073467MCP
get_claim(id=1073467)