AgentX
A public agentic benchmark released by SemiAnalysis, designed to evaluate the performance of inference frameworks under real agent workloads, simulating typical agent traffic patterns such as multi-tool invocation, multi-turn context accumulation, and concurrent task orchestration.
Core Facts
Timeline (last 90 days)
The test used DeepSeek's off-peak pricing.
In the AgentX test, vLLM's P90 per-user throughput was 196 tokens/s.
In the AgentX test, vLLM's total throughput reached 954K tokens/s/GPU (including cached, input, and output tokens).
In the AgentX test, vLLM achieved revenue of $38.96 per GPU hour.
In SemiAnalysis's AgentX benchmark, vLLM achieved the best-ever result for DeepSeek-V4.1-Flash using real agentic workloads.
In SemiAnalysis's AgentX benchmark, an 8-GPU AMD MI355X achieved 7,567 tok/s per GPU in Kimi K3 model serving at an interactive latency of 50 tok/s/user.
There is currently no public information on the specific model, performance metrics, or throughput data of this TPUv7 submission.
The AgentX platform announced that it received its first TPUv7-based model submission, and that this submission was served via vLLM.
AgentX is a model submission and serving hosting platform responsible for model deployment and providing external services, requiring integration with multiple inference backends and hardware targets.
AgentX stated that it looks forward to more models being deployed on TPUs through AgentX.
6 more timeline events
All Facts (16)
SemiAnalysis's AgentX benchmark is specifically designed for AI Agent scenarios, with a core focus on decoding latency under single-user conditions rather than maximum throughput.
75%UnverifiedIn the AgentX test, vLLM's P90 per-user throughput was 196 tokens/s.
50%UnverifiedThe test used DeepSeek's off-peak pricing.
50%UnverifiedIn the AgentX test, vLLM achieved revenue of $38.96 per GPU hour.
50%UnverifiedIn the AgentX test, vLLM's total throughput reached 954K tokens/s/GPU (including cached, input, and output tokens).
50%UnverifiedIn SemiAnalysis's AgentX benchmark, vLLM achieved the best-ever result for DeepSeek-V4.1-Flash using real agentic workloads.
50%UnverifiedIn SemiAnalysis's AgentX benchmark, an 8-GPU AMD MI355X achieved 7,567 tok/s per GPU in Kimi K3 model serving at an interactive latency of 50 tok/s/user.
50%UnverifiedThere is currently no public information on the specific model, performance metrics, or throughput data of this TPUv7 submission.
50%UnverifiedAgentX is a model submission and serving hosting platform responsible for model deployment and providing external services, requiring integration with multiple inference backends and hardware targets.
50%UnverifiedAgentX stated that it looks forward to more models being deployed on TPUs through AgentX.
50%UnverifiedThe test was run in SemiAnalysis's AgentX benchmark environment.
50%UnverifiedThe industry has long lacked a standardized evaluation system for Agentic workloads, and AgentX fills this gap.
50%UnverifiedThe vLLM team used the publicly available Agentic benchmark AgentX, introduced by SemiAnalysis, for measurement.
50%UnverifiedThe longer an Agent runs and the more tool results and history accumulate, the more likely that the original task requirements and operational constraints will be lost during truncation and compression if everything is stuffed into the context; key constraints must be readable at every round of execution.
50%UnverifiedThe AgentX platform announced that it received its first TPUv7-based model submission, and that this submission was served via vLLM.
50%UnverifiedThe vLLM team published a technical blog post titled "vLLM x AgentX: Optimizing for Real-World Agentic Serving."
50%