HarmBench
A benchmark designed to evaluate the safety of large language models, focusing on assessing a model's ability to resist harmful requests. It provides a standardized framework for testing model robustness against a range of potentially dangerous or policy-violating inputs.
Timeline (last 90 days)
The adversarially trained SEMA-Auto's average ASR@1 on HarmBench is 13.8 percentage points higher than the previous SOTA.
Against GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5 and GPT-6-Astra, the adversarially trained SEMA-Auto achieves an average of 59.6% ASR@1 on HarmBench.
The evaluation setup includes three guard judges, HarmBench as an external evaluator, and standard utility benchmarks.
HarmBench is a standardized adversarial attack evaluation framework that provides a unified set of attack prompts, a target model interface, and success determination rules.
The results of multidimensional item response theory (MIRT) modeling show that HarmBench's scores strongly do not support the "single attribute" assumption; it simultaneously captures multiple distinct dimensions.
Differential item functioning (DIF) analysis finds that models from different developers show systematic differences in HarmBench scores when their "refusal ability" is the same.
HarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
Several evaluation benchmarks released in 2024, such as JailbreakBench and HarmBench, have begun to incorporate assessments of multi-turn conversation scenarios.
Common alignment evaluation benchmarks include TruthfulQA and HarmBench.
Current mainstream evaluation benchmarks such as MMLU, TruthfulQA, and HarmBench are mostly designed for single dimensions and lack systematic measurement of value interaction effects.
3 more timeline events
All Facts (12)
HarmBench is a red-teaming benchmark specifically targeting harmful content generation
70%UnverifiedThe adversarially trained SEMA-Auto's average ASR@1 on HarmBench is 13.8 percentage points higher than the previous SOTA.
50%UnverifiedAgainst GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5 and GPT-6-Astra, the adversarially trained SEMA-Auto achieves an average of 59.6% ASR@1 on HarmBench.
50%UnverifiedHarmBench is a standardized adversarial attack evaluation framework that provides a unified set of attack prompts, a target model interface, and success determination rules.
50%UnverifiedThe evaluation setup includes three guard judges, HarmBench as an external evaluator, and standard utility benchmarks.
50%UnverifiedThe results of multidimensional item response theory (MIRT) modeling show that HarmBench's scores strongly do not support the "single attribute" assumption; it simultaneously captures multiple distinct dimensions.
50%UnverifiedDifferential item functioning (DIF) analysis finds that models from different developers show systematic differences in HarmBench scores when their "refusal ability" is the same.
50%UnverifiedHarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
50%UnverifiedSeveral evaluation benchmarks released in 2024, such as JailbreakBench and HarmBench, have begun to incorporate assessments of multi-turn conversation scenarios.
50%UnverifiedCommon alignment evaluation benchmarks include TruthfulQA and HarmBench.
50%UnverifiedCurrent mainstream evaluation benchmarks such as MMLU, TruthfulQA, and HarmBench are mostly designed for single dimensions and lack systematic measurement of value interaction effects.
50%UnverifiedHarmBench is a standardized red-teaming evaluation framework created by a CMU team.
50%