ComFair
A configurable multi-attribute fairness auditing benchmark that converts single-attribute instances into multi-attribute instances through a paired augmentation design, used to measure bias and performance changes of LLMs under combinations of intersectional attributes.
Timeline (last 90 days)
ComFair is a configurable fairness benchmark that converts existing fairness data into 479,066 controlled two-attribute and multi-attribute instances and pairs each augmented instance with a task-equivalent source instance.
ComFair's paired design supports unified auditing covering four task types: recognition, selection, continuation, and dialogue.
For each multi-attribute instance, ComFair constructs a "task-equivalent" single-attribute source instance, i.e., the two are identical in input semantics, labels, and task instructions, differing only in the combination of sensitive attributes.
Experiments on ten LLMs reveal that discriminative tasks (such as recognition and selection), even when overall changes are small, expose shifts in two directions, while generative tasks (such as continuation and dialogue) show larger and directionally asymmetric shifts.
ComFair is not a model of the natural joint distribution but a controlled auditing tool.
All Facts (5)
ComFair is a configurable fairness benchmark that converts existing fairness data into 479,066 controlled two-attribute and multi-attribute instances and pairs each augmented instance with a task-equivalent source instance.
50%UnverifiedComFair's paired design supports unified auditing covering four task types: recognition, selection, continuation, and dialogue.
50%UnverifiedFor each multi-attribute instance, ComFair constructs a "task-equivalent" single-attribute source instance, i.e., the two are identical in input semantics, labels, and task instructions, differing only in the combination of sensitive attributes.
50%UnverifiedExperiments on ten LLMs reveal that discriminative tasks (such as recognition and selection), even when overall changes are small, expose shifts in two directions, while generative tasks (such as continuation and dialogue) show larger and directionally asymmetric shifts.
50%UnverifiedComFair is not a model of the natural joint distribution but a controlled auditing tool.
50%