tau-bench
An AI agent evaluation benchmark proposed by Stanford University and collaborating institutions, designed to assess the reliability of tool-augmented agents in real-world database interaction scenarios. It measures whether system states match expected outcomes and evaluates success rates across repeated task executions.
Timeline (last 90 days)
tau-bench is a multi-turn agent evaluation benchmark proposed by Sierra AI that uses an LLM as a user simulator for multi-turn dialogue and judges by policy constraint compliance, for example, booking the correct flight but violating cabin-class rules is judged as a failure.
The UserProxyBench paper was written by Ashish Jain and Armaan Sandhu, and adds an evaluation layer on top of tau-bench to measure whether the simulated user completes the task.
tau-bench was proposed by the Sierra team, covers retail and airline customer service scenarios, and uses the final state of the database as the success criterion.
The tau-bench benchmark focuses on whether an agent reaches the expected database state and requires consistent success across multiple repeated attempts.
tau-bench is an agent evaluation benchmark proposed by Stanford University and other institutions, targeting the reliability assessment of tool-assisted agents in real database interaction scenarios.
tau-bench uses whether the system state after agent execution matches expectations as its criterion, rather than the output content of a single conversation.
All Facts (6)
tau-bench is a multi-turn agent evaluation benchmark proposed by Sierra AI that uses an LLM as a user simulator for multi-turn dialogue and judges by policy constraint compliance, for example, booking the correct flight but violating cabin-class rules is judged as a failure.
50%UnverifiedThe UserProxyBench paper was written by Ashish Jain and Armaan Sandhu, and adds an evaluation layer on top of tau-bench to measure whether the simulated user completes the task.
50%Unverifiedtau-bench was proposed by the Sierra team, covers retail and airline customer service scenarios, and uses the final state of the database as the success criterion.
50%UnverifiedThe tau-bench benchmark focuses on whether an agent reaches the expected database state and requires consistent success across multiple repeated attempts.
50%Unverifiedtau-bench is an agent evaluation benchmark proposed by Stanford University and other institutions, targeting the reliability assessment of tool-assisted agents in real database interaction scenarios.
50%Unverifiedtau-bench uses whether the system state after agent execution matches expectations as its criterion, rather than the output content of a single conversation.
50%