Compliance Rate
A metric in safety evaluation, defined as the proportion of model responses judged as compliant (fully providing harmful content) among all responses. It is sensitive to scorer bias, and differences between scorers can be amplified multiplicatively.
Timeline (last 90 days)
On two corpora each containing 900 responses and a total of eight scorers, the compliance rate differs by 22 to 25 times across scorers.
After excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.
Cross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.
All Facts (3)
On two corpora each containing 900 responses and a total of eight scorers, the compliance rate differs by 22 to 25 times across scorers.
50%UnverifiedAfter excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.
50%UnverifiedCross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.
50%