Class Collapse
A phenomenon in which a multi-class scorer systematically assigns samples of one class to an adjacent class near the decision boundary. It is a systematic bias of the scorer rather than random error, and it causes evaluation metrics to be structurally overestimated or underestimated.
Timeline (last 90 days)
The class collapse phenomenon of scorers also appeared with a 7B scorer from a fourth family, new harmful requests, and target models from a third family.
Class collapse refers to a scorer systematically pushing samples of one class toward an adjacent category near the decision boundary, rather than making random misjudgments.
Looking at the label count per category can serve as a zero-cost diagnostic for detecting scorer class collapse: if the compliant category count is abnormally low or even zero while adjacent categories are abnormally inflated, it indicates the scorer may be collapsing.
Even when each scorer is called only once, the results of scorer disagreement remain unchanged; collapse is not statistical fluctuation from multiple sampling.
All Facts (4)
The class collapse phenomenon of scorers also appeared with a 7B scorer from a fourth family, new harmful requests, and target models from a third family.
50%UnverifiedClass collapse refers to a scorer systematically pushing samples of one class toward an adjacent category near the decision boundary, rather than making random misjudgments.
50%UnverifiedLooking at the label count per category can serve as a zero-cost diagnostic for detecting scorer class collapse: if the compliant category count is abnormally low or even zero while adjacent categories are abnormally inflated, it indicates the scorer may be collapsing.
50%UnverifiedEven when each scorer is called only once, the results of scorer disagreement remain unchanged; collapse is not statistical fluctuation from multiple sampling.
50%