ComFair: A Multi-Attribute Joint Fairness Benchmark Revealing Intersectional Harms in LLMs

Why Is Single-Attribute Fairness Auditing Insufficient?

Fairness evaluations of large language models (LLMs) typically audit one sensitive attribute at a time—for example, focusing only on gender or only on race. This single-attribute paradigm can miss intersectional harms: when multiple attributes interact, non-additive effects produce failure modes that are invisible in marginal evaluations. According to the paper Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs, most existing benchmarks adopt single-attribute auditing and cannot capture model behavioral biases under joint multi-attribute distributions.

Intersectionality was first introduced by Kimberlé Crenshaw in 1989 to describe the compounded disadvantage Black women face in employment and legal contexts that cannot be explained by "race discrimination" or "sex discrimination" alone. In machine learning fairness research, intersectional harm refers to model behavior on joint values of multiple sensitive attributes (e.g., "female + ethnic minority"), which cannot be predicted by simply adding or averaging the model's marginal behavior on each attribute. Its mathematical root lies in the non-additivity of the model's decision function with respect to attribute combinations: if the model's bias toward gender and race contains an interaction term, the average bias measured by single-attribute auditing may be zero while specific intersectional subgroups suffer severe bias. Therefore, marginal auditing is statistically equivalent to projecting the joint distribution and inevitably loses interaction information—which is precisely why multi-attribute auditing is methodologically irreplaceable.

ComFair: A Configurable Multi-Attribute Fairness Benchmark

To address this problem, the researchers propose ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled two-attribute and multi-attribute instances, pairing each augmented instance with a task-equivalent source instance. This paired design enables unified auditing across four task types—recognition, selection, continuation, and dialogue—and can measure changes in performance or bias scores, the frequency with which added attributes introduce new errors or mask existing errors, and the stability of model rankings.

Core Design: Paired Augmentation and Task Equivalence

The key innovation of ComFair is that every multi-attribute instance corresponds to a single-attribute source instance, thereby isolating the causal effect of attribute combinations. This controlled experimental design allows researchers to precisely quantify the impact of intersectional fairness, rather than merely observing correlations in natural joint distributions.

Paired design draws on the ideas of matching and randomized controlled trials from causal inference. ComFair constructs a "task-equivalent" single-attribute source instance for each multi-attribute instance—that is, the two are consistent in input semantics, labels, and task instructions, differing only in the sensitive attribute combination. In this way, differences in model outputs on paired instances can be attributed to the attribute combination itself rather than confounders such as prompt length, vocabulary distribution, or task difficulty. Compared with observational auditing, paired design transforms "correlation comparison" into "controlled counterfactual comparison," thereby distinguishing two effects: whether the added attribute introduces new errors (false positive direction) and whether it masks existing errors (false negative direction). This decomposable error analysis is key to measuring the severity of intersectional harms and is also the basis for testing the stability of model rankings.

Experimental Findings: Task- and Subgroup-Dependent Sensitivity

Experiments on ten LLMs reveal task- and subgroup-dependent sensitivity. Discriminative tasks (such as recognition and selection) expose shifts in both directions even when overall changes are small; generative tasks (such as continuation and dialogue) exhibit larger and directionally asymmetric shifts; and model rankings change with attribute composition. These findings indicate that conclusions drawn from single-attribute auditing may not generalize robustly to multi-attribute settings.

Discriminative Tasks vs. Generative Tasks

In discriminative tasks, adding an attribute may simultaneously introduce new errors and mask old errors, but the net effects may cancel each other out, leaving overall metrics with modest changes. Generative tasks are more prone to directional shifts—for example, certain attribute combinations systematically degrade generation quality.

The difference between discriminative and generative tasks in fairness auditing stems from the structure of the output space. Discriminative tasks (recognition, selection) output discrete labels, and bias manifests as label flips or drift near probability thresholds; positive and negative errors may cancel out in overall accuracy, making aggregate metrics insensitive to attribute combinations. Generative tasks (continuation, dialogue) output free text, and bias is expressed along continuous dimensions such as lexical choice, sentiment tendency, and stereotype density. Without a natural "correct label" to offset deviations, attribute combinations are more likely to cause directionally consistent systematic shifts. This also explains why fairness evaluation of generative tasks requires more fine-grained metrics (such as sentiment scores, toxicity scores, and stereotype ratings) rather than relying solely on aggregate accuracy or error rates.

Implications for Fairness Evaluation

ComFair provides a scalable method for controlled sensitivity auditing rather than simulating natural joint distributions of identities. It emphasizes testing the robustness of fairness benchmark conclusions beyond single-attribute settings. For AI researchers and product teams, this means that evaluating model fairness requires joint multi-attribute auditing to avoid missing intersectional harms.

Limitations and Future Directions

The paper notes that ComFair is not a natural joint distribution model but a controlled auditing tool. Future work can extend to more attribute types and task scenarios and validate against real distributions.

In summary, multi-attribute fairness auditing is an important complement to LLM evaluation, and ComFair provides a configurable, scalable benchmark for this direction.

Controlled auditing and natural joint distributions are two complementary evaluation paradigms. Natural joint distribution auditing reflects deployment risks under real demographic compositions, but strong correlations between attributes (such as race and income, gender and occupation) make it difficult to separate the causal contribution of a single attribute. Controlled auditing breaks these correlations by artificially constructing attribute combinations and can cleanly identify interaction effects, but the real-world frequency of its attribute combinations may not match the weights in the benchmark. Therefore, ComFair's conclusions should be understood as "sensitivity detection" rather than "risk estimation": it answers which intersectional subgroups a model may fail on, not what proportion of deployment those subgroups represent. In the future, combining the two could use controlled auditing to locate vulnerable subgroups and then weight them with natural distribution data to obtain risk measures closer to reality.