Does LLM-Optimized Code Actually Consume More Energy? Only 2 of 44 Configurations Save Power in Testing

Using large models to automatically optimize code performance sounds both efficient and eco-friendly: if the code runs faster, it should naturally use less power. But an empirical study covering all 1,000 problems in EffiBench overturns this intuition. After measuring hardware-level energy consumption across 11 open-source models and 4 decoding configurations, the research team found that the "performance-optimized code" produced by LLM self-refinement generally consumes more energy than human engineers' reference implementations—42 of 44 configurations used more energy, with excess ranging from +4% to +886%.

An assumption long treated as true

"Faster code = less energy" is a rule of thumb long used in engineering, and LLM code-optimization pipelines are built on this premise. The study directly tested two things: whether performance-optimized code generated by LLM self-refinement is also more energy-efficient relative to human reference solutions; and whether the energy consumed by the optimization process itself can be offset by savings from subsequent runs.

Measurements covered all 1,000 problems in EffiBench, collecting hardware energy data at the process level for each run rather than relying on theoretical model estimates. This means the conclusions are based on real power curves, not coarse-grained estimates such as "CPU time × average power."

EffiBench is a benchmark for evaluating the efficiency of code generated by large models. It contains 1,000 programming problems, each paired with a performance-optimized human reference implementation and annotated with efficiency metrics such as execution time and memory usage. Unlike benchmarks that only examine functional correctness (such as HumanEval), EffiBench requires models to approach the resource consumption level of the reference solution while passing test cases. Researchers often use it to measure whether LLMs truly have the ability to "write efficient code," and this article uses its complete problem set to extend the evaluation dimension from runtime further to hardware energy consumption.

Inference energy: from 250 to 2,800 GPT-4.1 queries

The cost of the optimization process itself deserves attention first. Across different model and decoding configuration combinations, inference energy ranged from 0.79 MJ to 8.68 MJ, roughly equivalent to the energy of 250 to 2,800 GPT-4.1 queries. For teams that treat LLM optimization as part of daily development, this order of magnitude means the optimization activity itself leaves a considerable carbon footprint.

One immediately effective lever is limiting the maximum number of output tokens: compared with the default configuration, energy consumption dropped by up to 69%. This is intuitive—shorter outputs mean fewer decoding steps—but it also suggests that many optimization pipelines have a great deal of "redundant thinking" under default parameters.

Decoding configuration refers to a set of sampling parameters that affect the text generation process of a large model, commonly including temperature, top-p, top-k, beam search width, and maximum output tokens. The higher the temperature and the more random the sampling, the more likely the model is to "take detours" and generate long reasoning chains; beam search maintains multiple candidate paths in parallel, multiplying computation. Therefore, even for the same model, inference energy can differ by an order of magnitude under different decoding configurations. The 4 decoding configurations measured in this article are precisely intended to separate the contributions of "the model itself" and "the generation strategy" to energy consumption.

Execution-side reversal: 42/44 configurations use more energy

The truly counterintuitive part is the execution stage. Among the 44 model-configuration combinations, 42 generated code that consumed more energy than the human reference solution on every problem, with additional overhead extending from +4% to +886%. The only two exceptions offered merely marginal gains, and their break-even point requires on the order of 10^6 to 10^7 executions—almost no real-world project reaches this scale.

In other words, the kind of code that LLMs pursue as "faster on benchmarks" often trades higher memory usage, more aggressive parallelism, or more complex control flow for speed, at the cost of higher actual power consumption. Performance metrics and energy metrics systematically diverge here.

Architecture choice matters more than decoding parameters

The study also offers a finding with more guidance for sustainability practice: at the same parameter scale, mixture-of-experts (MoE) architectures have significantly lower inference energy than dense models, without sacrificing correctness. By contrast, the gains from tuning decoding parameters are local and limited.

This pushes the "green AI" discussion from "how to tune parameters" to "how to choose models"—decisions at the model architecture level have a more profound impact on long-term energy consumption than fine-tuning decoding configurations.

Mixture-of-Experts (MoE) architecture splits the feed-forward network into multiple "expert" subnetworks and, through a gating mechanism, activates only a few experts for each token. In this way, the total number of model parameters can be very large, but the parameters actually involved in computation in a single forward pass (active parameters) are far fewer than in dense models, significantly reducing compute and energy demands during inference. By contrast, every token in a dense model must be computed through all parameters. At comparable parameter scales, MoE gains an energy advantage precisely through "sparse activation" while maintaining correctness performance close to that of dense models.

Three takeaways for developers

Do not assume performance optimization equals energy savings. If sustainability is one of the project goals, energy consumption needs to be included as an evaluation metric rather than looking only at runtime or benchmark scores.

Set an output budget for optimization pipelines. Limiting output tokens can deliver up to 69% reduction in inference energy, making it one of the lowest-cost improvements.

Treat architecture selection as an environmental lever. While satisfying correctness requirements, the energy advantage of MoE over dense models deserves priority consideration more than repeatedly tuning decoding parameters.

The research team has made all artifacts public (DOI: 10.5281/zenodo.23598170), and the measurement pipeline on EffiBench is reproducible. The core reminder of this work is: LLM self-refinement is good at producing code that "looks faster," but reference solutions written by human engineers remain a difficult benchmark to surpass in terms of energy efficiency.

FAQ

Does LLM-optimized code always consume more energy? Not absolutely, but in the measured 44 configurations, 42 used more energy, and only 2 had marginal gains with an extremely high break-even point.

How can the energy consumption of LLM code optimization be reduced? Limiting the maximum output tokens can cut inference energy by up to 69%; choosing MoE architecture is more effective than tuning decoding parameters.

How broad is the coverage of this study? 11 open-source models, 4 decoding configurations, and all 1,000 problems in EffiBench, with hardware-level energy measurement throughout.

One-line Summary

In testing 44 LLM code-optimization configurations, the code produced by 42 of them actually consumed more energy than human reference solutions.

Paragraph Summary

An empirical study covering all 1,000 problems in EffiBench measured hardware-level energy consumption across 11 open-source models and 4 decoding configurations. The results show that "performance-optimized code" produced by LLM self-refinement generally consumes more energy than human engineers' reference implementations: 42 of 44 configurations used more energy, with excess ranging from +4% to +886%. The inference energy of the optimization process itself ranged from 0.79 to 8.68 MJ, equivalent to 250 to 2,800 GPT-4.1 queries. The study also found that limiting the maximum output tokens can cut inference energy by up to 69%, and that at the same scale, MoE architectures are more energy-efficient than dense models. The core conclusion is that performance optimization and energy efficiency are not naturally aligned, and developers should include energy consumption in evaluation metrics.