Unverified50% confidenceTradeoffExact time
Cross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedEven when each scorer is called only once, the results of scorer disagreement remain unchanged; collapse is not statistical fluctuation from multiple sampling.74% similarUnverifiedAfter excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.74% similarUnverifiedOn two corpora each containing 900 responses and a total of eight scorers, the compliance rate differs by 22 to 25 times across scorers.73% similarUnverifiedPaired comparison is more reliable than independent evaluation because it eliminates randomness introduced by sample selection.73% similarUnverifiedPercentage tolerance makes inputs of different magnitudes comparable and decouples thresholds from numerical scale, but the cost is that small absolute fluctuations in small numbers are amplified into high percentages, while large absolute changes in large numbers may only produce moderate percentages.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074742API
curl https://kongchang.com/api/v1/knowledge/claims/1074742MCP
get_claim(id=1074742)