Web Arena
An AI agent evaluation benchmark that packages target websites into fixed Docker images to assess agents' ability to complete tasks in web environments, ensuring reproducibility of evaluations.
Timeline (last 90 days)
CMU Graham Lab's Web Arena in 2024 populated environments with real website data scraped from Amazon, Reddit, and others; it was landmark work containing approximately 800 tasks across four applications, with validators hand-written by CMU graduate students.
Most mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
All Facts (2)
CMU Graham Lab's Web Arena in 2024 populated environments with real website data scraped from Amazon, Reddit, and others; it was landmark work containing approximately 800 tasks across four applications, with validators hand-written by CMU graduate students.
50%UnverifiedMost mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
50%