τ-bench
An evaluation benchmark targeting LLM tool-use and task-planning capabilities, with emphasis on measuring a model's reasoning and execution performance in complex tool-call chains and multi-step planning scenarios.
Timeline (last 90 days)
The paper evaluated GEPA-Curator on four agent benchmarks: GoBrowse, ALFWorld, AppWorld, and τ-bench.
Agent capability is usually measured through dimensions such as tool-calling accuracy, multi-step task completion rate, and instruction-following degree, with typical benchmarks including BFCL, τ-bench, and GAIA; Coding capability is measured through code generation and repair benchmarks (such as SWE-bench and LiveCodeBench).
τ-bench is an agent evaluation benchmark proposed by the Sierra team, used to measure a model's policy compliance in "tool-agent-user" three-party interaction scenarios, covering two domains: airline customer service and retail e-commerce customer service.
In its airline refund tool, τ-bench adopts a two-layer design: the tool description layer writes the policy requiring the model to first fetch the order and explicitly fill in expected_* parameters for reasoning; the server layer validates facts such as cabin class, insurance, and order time, with data coming from the database and server clock, not relying on parameters reported by the model.
τ-bench's criterion is: even if the agent successfully helps the user refund a ticket, as long as the refund action violates policy, the task is still judged as failed, i.e., it measures "task completion under the premise of following the rules".
Each task in τ-bench comes with a domain policy document specifying the conditions for refunds, rebooking, and exchanges; the agent must satisfy the user's requests across multiple turns without violating policy; evaluation metrics include task success rate and policy violation rate.
In the τ-bench retail probe, the action model predicted the environment's rejections with precision and recall of 1.000 on 11,763 held-out transitions, abstaining in only 0.26% of cases.
In the τ-bench retail environment benchmark test (30 test tasks, simulated customers, single run, solvi 0.8.0, gpt-oss-120b as the Agent), without the guard 18 tasks were solved and the environment rejected 10 mutating calls; with the guard 14 tasks were solved and the environment rejected 0 mutating calls.
τ-bench is a benchmark for tool-calling Agents proposed by the Sierra team, simulating retail and airline customer service scenarios, evaluating multi-turn dialogue, tool use and policy compliance, with the environment returning rejections for violating calls.
SWE-bench tests autonomous code repair capability, GAIA tests real-world multi-step task reasoning, and τ-bench tests tool-calling accuracy.
5 more timeline events
All Facts (12)
The paper evaluated GEPA-Curator on four agent benchmarks: GoBrowse, ALFWorld, AppWorld, and τ-bench.
50%UnverifiedAgent capability is usually measured through dimensions such as tool-calling accuracy, multi-step task completion rate, and instruction-following degree, with typical benchmarks including BFCL, τ-bench, and GAIA; Coding capability is measured through code generation and repair benchmarks (such as SWE-bench and LiveCodeBench).
50%Unverifiedτ-bench is an agent evaluation benchmark proposed by the Sierra team, used to measure a model's policy compliance in "tool-agent-user" three-party interaction scenarios, covering two domains: airline customer service and retail e-commerce customer service.
50%UnverifiedEach task in τ-bench comes with a domain policy document specifying the conditions for refunds, rebooking, and exchanges; the agent must satisfy the user's requests across multiple turns without violating policy; evaluation metrics include task success rate and policy violation rate.
50%Unverifiedτ-bench's criterion is: even if the agent successfully helps the user refund a ticket, as long as the refund action violates policy, the task is still judged as failed, i.e., it measures "task completion under the premise of following the rules".
50%UnverifiedIn its airline refund tool, τ-bench adopts a two-layer design: the tool description layer writes the policy requiring the model to first fetch the order and explicitly fill in expected_* parameters for reasoning; the server layer validates facts such as cabin class, insurance, and order time, with data coming from the database and server clock, not relying on parameters reported by the model.
50%Unverifiedτ-bench is a benchmark for tool-calling Agents proposed by the Sierra team, simulating retail and airline customer service scenarios, evaluating multi-turn dialogue, tool use and policy compliance, with the environment returning rejections for violating calls.
50%UnverifiedSWE-bench tests autonomous code repair capability, GAIA tests real-world multi-step task reasoning, and τ-bench tests tool-calling accuracy.
50%UnverifiedBenchmark environments targeting agent capabilities, such as WebArena, AgentBench, and τ-bench, still rely primarily on sandbox simulations and struggle to capture the real data noise, permission boundaries, latency jitter, and error propagation chains found in production-grade systems.
50%Unverifiedτ-bench is a benchmark released by Sierra AI for evaluating the reliability of conversational AI agents in real-world business scenarios.
50%Unverifiedτ-bench simulates customer service tasks in industries such as retail and aviation, using task pass rate as the core metric.
50%UnverifiedAgent evaluation benchmark suites include GAIA, AgentBench, and SWE-bench.
50%