Unverified50% confidenceBenchmarkExact time
After excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedRewriting one criterion in a prompt causes the pass rate of other criteria within the same prompt to fluctuate by 5 to 10 percentage points.76% similarUnverifiedIn the article, under DeepEval's default mode, 7 responses had their pass/fail conclusion flip at least once across three re-runs; after manually fixing evaluation_steps, the number of flips dropped to 3, and the agreement rate rose from 31/40 to 35/40.75% similarUnverifiedAt the 0.81 threshold, the proportion of cached answers that "fit the question" is 5.8 to 10.7 percentage points lower than fresh answers; converted to all responses, the overall quality loss is about 2 to 3 percentage points; Haiku 5.5 has the largest gap (10.7 percentage points), with a cached-answer pass rate of only 67.9%.75% similarUnverifiedAblation analysis shows that fine-tuning on the segment paradigm yields a +5.29-point improvement on the test set, while replacing the first 4,500 transcript words with 2,000 retrieved words yields only a +1.55-point improvement.75% similarUnverifiedCross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074740API
curl https://kongchang.com/api/v1/knowledge/claims/1074740MCP
get_claim(id=1074740)