Unverified50% confidenceBenchmarkExact time
On two tasks, biomedical inference and vulnerability detection, tested against 18 models, the best-of-10 attack flipped correct answers for an average of 54.9% of cases on proprietary models and reached 80.0% on open-weight models.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe benchmark produces two numbers for each model that are never merged: a task score and a false-confidence rate (the proportion of cases that should have been escalated but were answered anyway).75% similarUnverifiedRanked by question difficulty, persistence rate was 91.4% on the 60 questions that all models answered correctly, but only 73.5% on the 52 questions that at least one model got wrong, indicating that sycophancy tracks uncertainty.75% similarUnverifiedA study published in Communications Medicine tested 20 AI models across more than 10 million responses; after a brief safety reminder was added, potentially harmful clinical choices fell from 16.6% to 10.1%, with declines in 19 of the 20 models.74% similarUnverifiedIn Agent scenarios, directly comparing the average success rates of two groups of runs is easily misleading, and a more robust approach is to use a paired experimental design.72% similarVerifiedIn the experimental design, the AI used an 8:2 split to reserve 20% as an independent test set, and applied 5-fold cross-validation on the 80% training set to avoid data leakage.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074746API
curl https://kongchang.com/api/v1/knowledge/claims/1074746MCP
get_claim(id=1074746)