AgentBench
A comprehensive benchmark for evaluating AI agents across multiple operating environments, using single-task success rate as the core metric.
Core Facts
Timeline (last 90 days)
AgentBench and WebArena are examples of static benchmarks, providing reproducible test sets but prone to overfitting.
Industry Agent evaluation is shifting from static datasets to dynamic, interactive environment-based evaluation, such as AgentBench and WebArena, focusing on the decision quality and safety of models in real tool environments.
Most test scenarios in AgentBench and ToolBench are still pre-scripted, with a significant gap from the uncertainty of tool return values and diversity of errors encountered in real production environments
Benchmark environments targeting agent capabilities, such as WebArena, AgentBench, and τ-bench, still rely primarily on sandbox simulations and struggle to capture the real data noise, permission boundaries, latency jitter, and error propagation chains found in production-grade systems.
AgentBench是专门针对Agent的基准测试集
WebArena, ToolBench, and AgentBench are the current mainstream AI Agent benchmarks, focusing on web navigation, tool invocation, and multi-environment comprehensiveness, respectively.
In tasks requiring more than 5 steps, the success rate of the strongest models drops sharply from over 80% at the first step to 40% or even lower at subsequent steps, exhibiting an exponential decline.
AgentBench was proposed in 2023 by a research team from Tsinghua University and other institutions, with the paper published at ICLR 2024; it is the first multi-dimensional benchmark to systematically evaluate LLMs' capabilities as agents.
Agent evaluation benchmark suites include GAIA, AgentBench, and SWE-bench.
All Facts (10)
AgentBench covers 8 types of environments including operating systems, databases, and games, and comprehensively evaluates an Agent's multi-domain adaptability.
65%UnverifiedAgentBench and WebArena are examples of static benchmarks, providing reproducible test sets but prone to overfitting.
50%UnverifiedIndustry Agent evaluation is shifting from static datasets to dynamic, interactive environment-based evaluation, such as AgentBench and WebArena, focusing on the decision quality and safety of models in real tool environments.
50%UnverifiedMost test scenarios in AgentBench and ToolBench are still pre-scripted, with a significant gap from the uncertainty of tool return values and diversity of errors encountered in real production environments
50%UnverifiedBenchmark environments targeting agent capabilities, such as WebArena, AgentBench, and τ-bench, still rely primarily on sandbox simulations and struggle to capture the real data noise, permission boundaries, latency jitter, and error propagation chains found in production-grade systems.
50%UnverifiedAgentBench是专门针对Agent的基准测试集
50%UnverifiedWebArena, ToolBench, and AgentBench are the current mainstream AI Agent benchmarks, focusing on web navigation, tool invocation, and multi-environment comprehensiveness, respectively.
50%UnverifiedAgentBench was proposed in 2023 by a research team from Tsinghua University and other institutions, with the paper published at ICLR 2024; it is the first multi-dimensional benchmark to systematically evaluate LLMs' capabilities as agents.
50%UnverifiedIn tasks requiring more than 5 steps, the success rate of the strongest models drops sharply from over 80% at the first step to 40% or even lower at subsequent steps, exhibiting an exponential decline.
50%UnverifiedAgent evaluation benchmark suites include GAIA, AgentBench, and SWE-bench.
50%