SemiAnalysis AgentX
An evaluation benchmark from SemiAnalysis aimed at AI agent workloads. It focuses on throughput and cost performance in long-loop, multi-turn tool-calling scenarios and is closer to practical agent deployment than general-purpose dialogue benchmarks.
Timeline (last 90 days)
The vLLM team published DeepSeek-V4.1-Flash-1017's results on the SemiAnalysis AgentX benchmark: 1.3 billion tokens can be run per dollar of TCO.
On the SemiAnalysis AgentX benchmark, the model achieves single-card throughput of 830,000 tokens/s (including cache, input and output), based on an interaction speed of 100 tokens/s per user.
Benchmark numbers are worth referencing but cannot be directly extrapolated to production environments; it is advisable to re-measure throughput and cost under your own traffic patterns.
A single benchmark cannot represent all workloads; training, fine-tuning and ultra-long-context scenarios still require independent verification.
All Facts (4)
The vLLM team published DeepSeek-V4.1-Flash-1017's results on the SemiAnalysis AgentX benchmark: 1.3 billion tokens can be run per dollar of TCO.
50%UnverifiedOn the SemiAnalysis AgentX benchmark, the model achieves single-card throughput of 830,000 tokens/s (including cache, input and output), based on an interaction speed of 100 tokens/s per user.
50%UnverifiedBenchmark numbers are worth referencing but cannot be directly extrapolated to production environments; it is advisable to re-measure throughput and cost under your own traffic patterns.
50%UnverifiedA single benchmark cannot represent all workloads; training, fine-tuning and ultra-long-context scenarios still require independent verification.
50%