ServerBench: Can Coding Agents Build Their Own LLM Inference Serving Systems?

LLM coding agents are moving from "writing code" to "building systems." ServerBench poses a bold question: can agents design the inference serving systems that power themselves? This benchmark pushes agents into the role of systems engineers, requiring them to make a full set of implementation and operational decisions under latency constraints, session dependencies, and correctness requirements.

What Problem Does ServerBench Solve?

Traditional evaluations of coding agents focus mostly on function-level or repository-level code generation, but an LLM inference serving system is a classic systems engineering problem: it involves batching strategies, cache management, request routing, resource allocation, and many other coupled components. These decisions constrain one another—improving throughput may sacrifice latency, and adding cache may eat into GPU memory.

ServerBench's core innovation is turning "end-to-end system design" into a quantifiable experimental task. The agent does not face a programming problem with a standard answer, but an open environment that requires repeated experimentation, observation of feedback, and strategy adjustment. It must find balance points within constraints, just like a real systems engineer.

Core Metric: How Many Latency-Compliant Tokens per Dollar

ServerBench designs a straightforward but stringent objective function: dollars per dollar, i.e., the value of latency-compliant tokens produced per dollar of GPU cost. This metric compresses three dimensions at once—cost, performance, and quality—so no single-point optimization can slip through.

The benchmark uses stateful replay to preserve real session dependencies, ensuring that the request stream the agent faces is not isolated but contextually coherent. Submitted systems also undergo independent verification to prevent agents from "cheating" their way through tests. This design makes evaluation results closer to real deployment scenarios.

Stateful replay is an evaluation method: it replays not only the content of requests but also the session state between requests (such as conversation history, cache keys, and dependencies), ensuring that the agent faces a contextually coherent real request stream. Compared with stateless replay, stateful replay can expose system behaviors such as cache hit rates and session-sticky routing, making it closer to production environments.

Independent verification means that after the agent submits a system, the testers use inspection methods isolated from the agent's development environment (such as unit tests, runtime monitoring, and input mutation testing) to confirm that the system actually executes the expected code paths, rather than "passing" tests through hardcoding, cache cheating, or bypassing key logic.

Experimental Results: Claude Sonnet 5.5 Stands Out

According to the paper's data, across 11 coding agent models and three deployment configurations, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B, reaching 4.87 times the vLLM reference implementation. Of 33 runs, 22 produced verified systems, and 13 of those surpassed the strongest baseline.

These numbers show two things: first, leading coding agents already have the ability to design usable inference systems; second, the gap between agents is considerable, and not all models can deliver reliably. vLLM, as a mature open-source inference framework, was substantially surpassed by agent solutions, suggesting that specialized designs do have room for optimization under specific task specifications.

vLLM is an open-source high-throughput LLM inference and serving engine whose core mechanisms include PagedAttention and continuous batching, and it is widely used as a reference implementation for inference serving in general scenarios. Using it as the baseline in ServerBench means that agents need to specialize their designs for specific task specifications (such as fixed latency constraints and particular session patterns) in order to surpass this general framework.

Three Key Findings

The paper's analysis distills three valuable observations:

High-value designs jointly optimize capacity and load selection. High-performing agents do not adjust a single parameter in isolation; they consider capacity planning (such as batch size and GPU memory allocation) together with load selection (such as request priority and routing strategy). This coordinated thinking reflects systems design capability.

Agents need workload-relevant experiments to interpret performance feedback. Merely seeing latency numbers change is not enough; agents must design experiments that reveal causal relationships in order to understand the reasons behind the feedback. This requires agents to have experimental design capability, not just passive parameter tuning.

Reliable delivery requires verification of the execution paths introduced by optimization. Agents may introduce new code paths during optimization, and whether these paths are actually executed and correct requires additional verification. Otherwise, a system may perform well in testing but expose problems during actual operation.

Execution path verification means confirming, after system optimization, that newly added or modified code branches are triggered and execute correctly at runtime. When tuning parameters or refactoring, agents may introduce conditional branches, cache shortcuts, or concurrency logic; if these paths are not covered by tests, the system may "luckily" pass benchmark tests because they are not triggered, yet fail under real workloads. Therefore, runtime instrumentation, path coverage analysis, or differential testing is needed for verification.

Implications for Agent System Design

ServerBench's significance goes beyond a leaderboard. It makes the question "can agents do systems engineering" measurable, reproducible, and comparable. For researchers and engineers, this benchmark provides a window into how agents translate serving objectives and feedback into effective systems.

From a broader perspective, as coding agents begin designing the inference systems that power themselves, a recursive self-optimization loop is forming. Agents improve systems, and better systems in turn provide stronger foundations for agents to run on. ServerBench provides the first systematic experimental platform for studying this loop.

One-line Summary

ServerBench evaluates whether coding agents can design cost-effective LLM inference serving systems.

Paragraph Summary

ServerBench pushes coding agents from code generation toward end-to-end system design, requiring agents to autonomously make decisions about batching, caching, routing, and other components under latency constraints, session dependencies, and correctness requirements. The benchmark adopts a dollars-per-dollar metric (the value of latency-compliant tokens produced per dollar of GPU cost) and uses stateful replay and independent verification to ensure realism and prevent cheating. Experiments show that Claude Sonnet 5.5 reaches 4.87 times the vLLM reference implementation on Gemma E4B, with 22 of 33 runs passing verification. The paper distills three key findings: jointly optimizing capacity and load selection, interpreting feedback through workload-relevant experiments, and verifying execution paths introduced by optimization. ServerBench provides the first systematic experimental platform for studying the agent self-optimization loop.