ToolTurns: A Multi-Turn Conversation Tool-Use Benchmark Where the Strongest Model Achieves Only 39.8% Success Rate

Microsoft's ToolTurns benchmark reveals agents' multi-turn tool-use gap: nearly 70% single-request success but under 40% full-conversation success.
Microsoft Research and other institutions released ToolTurns, a multi-turn tool-use benchmark with 2,620 questions, 1,827 tools, and 11,004 API endpoints across 47 categories. Each conversation begins with a deliberately incomplete request plus two context-dependent follow-ups. The best agent solves 69.6% of single requests but only 39.8% of full conversations, and all models do worse on follow-ups. An open-source user simulator with a checker-fixer pipeline improves dialogue quality.
Calling tools within multi-turn conversations has long been a key bottleneck for deploying agents in practice. Most existing large-scale tool-use benchmarks focus on isolated single requests, whereas in real-world scenarios, user needs are often vague and require gradual clarification, and conversation context must be accurately carried forward. The ToolTurns benchmark, recently released by Microsoft Research and other institutions, is designed to fill this gap.

What is ToolTurns?
ToolTurns is a multi-turn conversation tool-use benchmark containing 2,620 questions, covering 1,827 tools and 11,004 API endpoints across 47 categories. Its core design is that each conversation starts with an initial request that is deliberately information-incomplete, followed by two context-dependent follow-up questions. This structure forces agents not only to learn how to call tools, but also to proactively clarify vague requirements and maintain conversation context consistency across multiple turns.
Unlike single-turn benchmarks, ToolTurns simulates the progressive interaction between a real user and an assistant. The initial request may lack key parameters (such as time range, location, or object), and the agent must either ask the user for clarification or make reasonable inferences based on information gradually revealed in the conversation.
Tool-use benchmark refers to a standardized test set for evaluating a language model's ability to call external tools (such as search, calendar, or database APIs). Single-turn benchmarks usually provide a complete and explicit request and test whether the model can select the right tool and fill in the right parameters, whereas multi-turn benchmarks require the model to handle missing information, evolving requirements, and context inheritance during a conversation. ToolTurns belongs to the latter category, and its "deliberately information-incomplete" design draws on the interaction pattern of real users progressively clarifying their needs.
API endpoint refers to a specific callable service interface. For example, "query the weather in a certain city on a certain day" corresponds to one endpoint, while "weather service" is a tool. ToolTurns covers 1,827 tools and 11,004 endpoints, meaning that a single tool may have multiple interfaces with different functions. The agent needs to precisely select the endpoint and pass parameters correctly, which is more challenging than merely selecting a tool.
Key Finding: Strong in Single-Turn, Weak in Multi-Turn
The paper evaluates nine language models, and the results reveal a common phenomenon: the best agent achieves a 69.6% solve rate on individual requests, but the full-conversation completion rate drops sharply to 39.8%. This means that even if a model can handle isolated tool calls well, maintaining task success in a multi-turn context remains very difficult.
More notably, under this evaluation protocol, all evaluated models have lower success rates on follow-up questions than on the initial request. This shows that the problem of models losing context or failing to effectively use prior information during a conversation is widespread rather than a defect of individual models.
Dialogue Quality and Task Success May Be Decoupled
The study also finds that dialogue-quality rubric scores and task success rates may rank models very differently. A model may generate fluent, natural conversations while performing poorly on tool-calling tasks, and vice versa. This suggests that when evaluating agents, we cannot look only at whether the final task is completed; the quality of the dialogue process also needs to be measured independently.
Solve rate usually refers to the proportion of single requests in which an agent successfully completes the task, while conversation completion rate requires all turns in a multi-turn conversation (the initial request plus follow-up questions) to succeed. The huge gap between the two (69.6% vs. 39.8%) shows that models perform reasonably well on isolated requests, but once they need to maintain state across turns, inherit prior information, and handle follow-up questions, the failure rate rises significantly. This evaluation protocol separates "single-point capability" from "sustained collaboration capability," bringing it closer to real deployment scenarios.
Open-Source User Simulator and Checker-Fixer Pipeline
To generate multi-turn conversations scalably, ToolTurns comes with an open-source user simulator. The simulator gradually reveals hidden context through dialogue and generates follow-up questions. To improve the quality of simulated conversations, the authors introduce a checker-fixer pipeline: it first detects defects in the conversation turns generated by the simulator, then corrects them.
Paired-turn evaluations show that the checker-fixer reduces detected dialogue defects, and blind human evaluations also prefer the corrected turns. This provides a reliable conversation-generation infrastructure for future research on multi-turn tool use.
User simulator is a component in which a language model plays the role of the user and gradually generates conversation turns based on a preset hidden context, used to synthesize multi-turn interaction data at low cost and at scale. It needs to simulate real users' vague expressions, follow-up questions, and clarification behavior rather than directly giving complete requirements.
Checker-fixer pipeline is a two-stage quality-assurance mechanism: the checker detects defects in simulated conversations (such as information leakage, logical contradictions, or repeated questions), and the fixer corrects them. This "generate-detect-correct" pipeline is often used to improve synthetic data quality. Blind human evaluations preferring the corrected turns indicate that the pipeline can effectively reduce noise and provide more reliable conversation samples for benchmarking.
Implications for Agent Development
The results of ToolTurns point in a clear direction: tool-calling capability cannot be validated only with single-turn benchmarks. In real deployments, user requests are often incomplete and requirements evolve during the conversation, so agents need the ability to proactively clarify, inherit context, and plan across turns.
For developers, this means that when training and evaluating agents, they should introduce more context-dependent follow-up scenarios rather than only static API-calling tests. At the same time, the decoupling of dialogue quality and task success also reminds us that a multi-dimensional evaluation system is needed to guide model iteration.
ToolTurns is open source, and its benchmark and user simulator provide the research community with a reproducible test platform for measuring whether agents can sustain task success across multi-turn tool conversations.
Conclusion
ToolTurns reveals a significant weakness of current LLM agents in multi-turn tool conversations: the single-request success rate is nearly 70%, while the full-conversation success rate is less than 40%. All models perform worse on follow-up questions, indicating that context maintenance is a common challenge. The accompanying open-source user simulator and checker-fixer pipeline provide tools for generating high-quality multi-turn conversation data. For teams focused on deploying AI agents, this benchmark is worth incorporating into their evaluation systems.
One-line Summary
Microsoft's ToolTurns benchmark reveals agents' weakness in multi-turn tool calling: nearly 70% single-request success rate, but less than 40% full-conversation success.
Paragraph Summary
Microsoft Research and other institutions released ToolTurns, a multi-turn tool-use benchmark containing 2,620 questions, 1,827 tools, and 11,004 API endpoints across 47 categories. Each conversation starts with a deliberately information-incomplete initial request, followed by two context-dependent follow-up questions, forcing agents to proactively clarify and maintain context consistency. Evaluating nine models finds that the best agent achieves a 69.6% single-request solve rate, but only a 39.8% full-conversation completion rate, and all models have lower success rates on follow-up questions than on the initial request. The study also finds that dialogue-quality scores and task success rates may produce different rankings. The accompanying open-source user simulator reduces dialogue defects through a checker-fixer pipeline, and blind human evaluations prefer the corrected turns.
Related articles

US Classified National Security Communications Network Outage: A Security Warning for the AI Era
Reuters reports a highly classified US national security communications network outage, sparking Hacker News discussion. This article analyzes critical infrastructure resilience challenges and their implications for AI system security design and failure recovery.

SEC Investigates Nvidia: Behind the Jensen Huang Departure Controversy
The Wall Street Journal reports the SEC is investigating Nvidia following CEO Jensen Huang's departure and admitted affair with an alleged Chinese spy. This article maps the timeline, compliance risks, and potential impact on the AI chip industry, while noting the information awaits cross-verification.

MacBook Neo Sees Rare $50 Discount Ahead of Black Friday — A Good Time for Students to Buy
MacBook Neo gets a rare $50 discount ahead of Black Friday: 256GB at $649, 512GB at $749, all four colors available at Amazon, Best Buy, and Target—great for students.