Review Consistency
The degree to which scores and comments on the same paper converge when multiple reviewers rate it independently; it is an important indicator of the quality of peer review at academic conferences and is commonly measured using inter-rater reliability coefficients such as standard deviation and Krippendorff's alpha.
Timeline (last 90 days)
Under non-blind review conditions (in the second and third rounds of review the previous scores and comments were visible), 1022 of the 1440 reviews were consistent judgments.
Under blind review conditions (with previous scores and comments hidden), the number of consistent judgments on the same set of data dropped to 749.
All Facts (2)
Under non-blind review conditions (in the second and third rounds of review the previous scores and comments were visible), 1022 of the 1440 reviews were consistent judgments.
50%UnverifiedUnder blind review conditions (with previous scores and comments hidden), the number of consistent judgments on the same set of data dropped to 749.
50%