CompressionBench: Can Coding Agents Autonomously Compress LLMs?

Introduction

Recursive self-improvement requires AI systems to be able to optimize the models and infrastructure used by subsequent systems. LLM compression provides a concrete efficiency target: preserving the behavior of a pretrained model within a smaller model storage budget. CompressionBench is built around this question—can coding agents autonomously complete this step?

According to the CompressionBench research, the team formalizes compression as a terminal task: the agent receives a source checkpoint, storage limit, inference environment, and time budget, and then builds compressed weights and executable inference code. The agent controls its own algorithms and experiments, perplexity feedback guides the search, and independent verification checks the final model.

Perplexity is a metric for measuring a language model's predictive capability, defined as the exponentiated average negative log-likelihood of the model on test text. Lower perplexity indicates that the model predicts text more accurately. In model compression, it is often used as a proxy metric to evaluate the behavioral gap between the compressed model and the original model. However, perplexity does not directly reflect downstream task performance, so independent verification and downstream accuracy are needed for a comprehensive evaluation.

BF16 is a half-precision floating-point format (Brain Floating Point 16) widely used in deep learning to accelerate inference and reduce memory usage. Compared with FP16, BF16 has a wider dynamic range and is suitable for both training and inference. In compression benchmarks, BF16 is typically used as the uncompressed baseline model for comparison against the performance loss after compression.

Compression as an Agent Terminal Task

Defining compression as a terminal task means the agent must deliver a runnable compressed model end to end, not merely propose ideas or generate code snippets. It needs to autonomously decide which compression algorithm to use, how to allocate the storage budget, how to tune hyperparameters, and actually run verification in the inference environment.

This setting extends the capability boundary of coding agents from "writing code" to "running experiments and delivering results." The agent must not only understand model structure, but also possess experiment design, result analysis, and iterative optimization capabilities.

Terminal Task in agent evaluation refers to a task in which the agent must autonomously complete the full process from input to output and deliver an executable result, rather than only providing intermediate artifacts or suggestions. In CompressionBench, the terminal task requires the agent to receive inputs such as the source checkpoint and storage limit, output compressed weights and inference code, and ensure the final model passes independent verification. This setting tests the agent's end-to-end problem-solving ability, including algorithm selection, experiment design, and result delivery.

Experimental Results: Success Rate and Performance Loss

Across six agents and six model-size settings, 29 of 36 submissions passed verification. This success rate shows that coding agents can complete the compression task in most cases, but about one-fifth of submissions still failed independent verification, exposing reliability issues in autonomous experimentation.

In terms of performance, when selecting the agent design with the lowest perplexity for each task, average perplexity increased by 3.4% over BF16, and downstream accuracy was 77.33%, compared with 79.56% for BF16. This gap indicates that agent-driven compression still has room for improvement in preserving model quality, but it can already achieve usable compression under storage budget constraints.

How Agents Iterate: Behaviors Revealed by Logged Experiments

The study logged the agents' experiment processes, revealing how they achieved these results and where improvements broke down. With storage nearly fixed, agents reduced perplexity in 21 of 24 revisions—showing that they can use feedback signals for effective search.

Full-model feedback reversed 17 of 79 prior local rankings, meaning that judgments based on local metrics alone can be misleading and that global verification is essential for correctly evaluating compression schemes. Generation and inference tests also exposed failure modes that likelihood alone cannot capture, suggesting that evaluating compressed models cannot rely on perplexity alone.

Local Ranking vs. Global Verification: During the agent search process, local ranking may rank candidate schemes based on a partial validation set or proxy metrics (such as perplexity). However, local ranking can be misleading due to overfitting or metric bias. Full-model feedback evaluates on the entire validation set and can reveal flaws not reflected in local rankings. CompressionBench found that full-model feedback reversed 17 of 79 local rankings, indicating that relying only on local metrics may lead to selecting suboptimal schemes, and global verification is essential to ensure the practical usability of compressed models.

Key Challenges and Implications

The results identify representation design, experimental judgment, and capability verification as core elements of agent-driven model compression. Representation design determines the boundaries of compression, experimental judgment affects search efficiency, and capability verification ensures that the compressed model is truly usable rather than merely looking good on metrics.

These findings position model compression as a measurable component of efficiency-oriented self-improvement. Coding agents have already demonstrated the potential to autonomously complete compression tasks, but there is still clear room for improvement in reliability, evaluation methods, and final quality. For researchers and engineers focused on AI self-improvement and coding agent applications, CompressionBench provides a reproducible benchmark and a clear direction for improvement.

One-line Summary

CompressionBench treats LLM compression as a terminal task for coding agents and evaluates their autonomous compression capabilities and limitations.

Paragraph Summary

CompressionBench studies whether coding agents can autonomously complete LLM compression tasks. The researchers formalize compression as a terminal task: the agent receives inputs such as the source checkpoint and storage limit, and outputs compressed weights and executable inference code. Across six agents and six model settings, 29 of 36 submissions passed verification, a success rate of about 80%. The best agent designs had average perplexity 3.4% higher than BF16 and downstream accuracy of 77.33% versus 79.56%. Agents reduced perplexity in 21 of 24 revisions, but full-model feedback reversed 17 of 79 local rankings, indicating that local metrics can be misleading. The study identifies representation design, experimental judgment, and capability verification as core challenges, providing a measurable benchmark for efficiency-oriented self-improvement.