Cohen's Kappa

When building a golden test set, samples need to cover various levels of judgment difficulty and edge cases, labels need to be independently annotated by multiple people with inter-annotator agreement calculated (such as Cohen's kappa), and labels need to be periodically reviewed for staleness due to changes in business rules.

Gwet's AC1 is a statistic used to measure inter-rater agreement, applicable to classification annotation tasks with imbalanced category distributions, and is more robust than Cohen's Kappa when one category accounts for an extremely high proportion.

Inter-rater agreement is a statistical metric measuring how consistently multiple raters score the same object, commonly quantified using Cohen's Kappa, Krippendorff's Alpha, or the intraclass correlation coefficient (ICC).

Sampling several hundred of the most contested 2+/1+ boundary cells and having 2–3 pathologists independently re-label them and calculating Cohen's Kappa or Fleiss' Kappa can reframe the problem from model underperformance to unclear label definitions.

Inter-rater reliability is commonly measured using Cohen's Kappa coefficient to ensure consistency and objectivity in classification standards

Traditional codebook development follows an iterative reliability testing process, repeatedly revising based on inter-rater agreement metrics (such as Cohen's Kappa or Krippendorff's Alpha) until agreement reaches an acceptable threshold (typically Kappa ≥ 0.7).

The audit results show that automatic labels agree with human consensus on 90.3% of claims, with a Cohen's Kappa coefficient of 0.87.

Even experienced human experts performing the same review task often achieve inter-group Kappa values of only 0.4–0.6.

The standard interpretation of Cohen's Kappa is typically: κ < 0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and above 0.80 almost perfect.

标注者间一致性IAA常用Cohen's Kappa系数（两个标注者）和Fleiss' Kappa系数（多个标注者）计算，Kappa值为1表示完全一致，0表示与随机一致无差异