EffiBench
A benchmark for evaluating the efficiency of code generated by large language models. It contains 1000 programming problems, each paired with a performance-optimized human reference implementation and annotated with efficiency metrics such as execution time and memory usage, and is used to measure the ability of LLMs to write efficient code.
Timeline (last 90 days)
An empirical study conducted hardware-level energy measurements on all 1000 EffiBench problems across 11 open-source models and 4 decoding configurations, finding that 42 of the 44 configurations generated code that was more energy-consuming than the human reference implementation.
EffiBench is a benchmark for evaluating the efficiency of code generated by large models, containing 1000 programming problems, each paired with a performance-optimized human reference implementation and annotated with efficiency metrics such as execution time and memory usage.
Unlike benchmarks that only examine functional correctness (such as HumanEval), EffiBench requires models to approach the resource consumption level of the reference solution as closely as possible while passing the test cases.
The research team has made all artifacts public (DOI: 10.5281/zenodo.23598170), and the measurement pipeline on EffiBench is reproducible.
All Facts (4)
An empirical study conducted hardware-level energy measurements on all 1000 EffiBench problems across 11 open-source models and 4 decoding configurations, finding that 42 of the 44 configurations generated code that was more energy-consuming than the human reference implementation.
50%UnverifiedEffiBench is a benchmark for evaluating the efficiency of code generated by large models, containing 1000 programming problems, each paired with a performance-optimized human reference implementation and annotated with efficiency metrics such as execution time and memory usage.
50%UnverifiedUnlike benchmarks that only examine functional correctness (such as HumanEval), EffiBench requires models to approach the resource consumption level of the reference solution as closely as possible while passing the test cases.
50%UnverifiedThe research team has made all artifacts public (DOI: 10.5281/zenodo.23598170), and the measurement pipeline on EffiBench is reproducible.
50%