Unverified50% confidenceFactExact time
Even with the judge's temperature set to 0, judging the same answers and chunks three times, the 18-question mean of precision fluctuated by 0.085, faithfulness by 0.042, recall by 0.046, and only relevancy stayed relatively stable at 0.003.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.76% similarUnverifiedIn the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.73% similarUnverifiedIn the example, for the same set of human labels (18 pass, 2 fail), a lazy judge that passes everything gets an accuracy of 0.90 but a kappa of 0.00; a better judge gets an accuracy of 0.95 but a kappa of only 0.77, still not reaching the 0.80 threshold.67% similarUnverifiedWhen only 5% of outputs fail, an evaluator that passes everything agrees with the expert 95% of the time, yet its recall is zero, which is called the "agreement paradox."67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1065190API
curl https://kongchang.com/api/v1/knowledge/claims/1065190MCP
get_claim(id=1065190)