BBQ
A bias benchmark dataset designed for question-answering scenarios. It evaluates model performance on social bias dimensions by constructing multiple-choice questions set in ambiguous contexts, covering social group attributes including race, gender, and age.
Timeline (last 90 days)
The confidence thresholds recommended by model manufacturers increased the proportion of stereotypes among retained answers on ambiguous BBQ gender questions, with granite-switch's stereotype proportion rising from 0.76 to 0.85 and pplx's from 0.81 to 1.00.
BBQ is a dataset specifically designed to evaluate social bias in question-answering models, covering multiple dimensions such as gender, race and age, with questions divided into ambiguous-context and unambiguous-context categories.
Elastic by default applies int8 quantization to float32 vectors to reduce memory usage, BBQ further compresses vectors to nearly one bit, and disk BBQ nearly eliminates the memory overhead of vector search.
Mainstream bias benchmarks currently include datasets such as WinoBias, StereoSet, and BBQ, which measure a model's stereotyping tendencies along dimensions such as gender and race by constructing contrastive sentence pairs or multiple-choice questions.
A common limitation shared by mainstream bias benchmarks is that they measure a model's explicit choices in specific controlled contexts, rather than implicit biases that emerge in open-ended generation, long-text reasoning, or complex role-playing scenarios.
BBQ (Bias Benchmark for QA) detects models' bias tendencies across nine social dimensions through ambiguous and disambiguating question-answer pairs.
All Facts (6)
The confidence thresholds recommended by model manufacturers increased the proportion of stereotypes among retained answers on ambiguous BBQ gender questions, with granite-switch's stereotype proportion rising from 0.76 to 0.85 and pplx's from 0.81 to 1.00.
50%UnverifiedBBQ is a dataset specifically designed to evaluate social bias in question-answering models, covering multiple dimensions such as gender, race and age, with questions divided into ambiguous-context and unambiguous-context categories.
50%UnverifiedElastic by default applies int8 quantization to float32 vectors to reduce memory usage, BBQ further compresses vectors to nearly one bit, and disk BBQ nearly eliminates the memory overhead of vector search.
50%UnverifiedMainstream bias benchmarks currently include datasets such as WinoBias, StereoSet, and BBQ, which measure a model's stereotyping tendencies along dimensions such as gender and race by constructing contrastive sentence pairs or multiple-choice questions.
50%UnverifiedA common limitation shared by mainstream bias benchmarks is that they measure a model's explicit choices in specific controlled contexts, rather than implicit biases that emerge in open-ended generation, long-text reasoning, or complex role-playing scenarios.
50%UnverifiedBBQ (Bias Benchmark for QA) detects models' bias tendencies across nine social dimensions through ambiguous and disambiguating question-answer pairs.
50%