Agent Evaluation

Each Agent run incurs actual API call costs, and as the scale of evaluation use cases grows and retries accumulate, costs can mount rapidly.

Agent evaluation differs from traditional software testing: traditional testing relies on preset test cases and its pass rate can be determined in one go, whereas Agent behavior is generated by a model in open contexts, making test cases difficult to enumerate exhaustively.

When evaluating Agent performance, Asana focuses on three metrics: completion rate in real workflows, frequency of human intervention, and users' adoption of Agent suggestions.

In Agent evaluation, controlled experiments require strict control of variables — apart from the factor under test, the environment's initial state, random seeds, opponent strategies, number of rounds, etc. should all be kept consistent.

Agent evaluation is more difficult than ordinary LLM applications because an agent makes a series of tool calls and decisions, and every step in the chain can go wrong.

Traditional evaluation metrics such as accuracy cannot characterize planning quality, tool-call correctness, or error-recovery capability, so a trajectory-level, multi-dimensional, reproducible evaluation framework is needed.

The complexity of Agent evaluation comes from the multi-turn nature of the tasks themselves, changes in environment state, and the irreversibility of Agent decisions.

Currently mainstream Agent evaluation methods include static evaluation based on benchmarks, interactive evaluation based on simulated environments, and subjective evaluation based on human feedback.

Agent evaluation is not a one-time acceptance action but a feedback loop that accompanies iteration.

The engineering lesson the authors drew from the experiment is that agent automation should record "automatically triggered actions" and "actively decided actions" separately, otherwise logs will misread defensive automation as aggressive intent.

Most Agent evaluations focus only on task completion (such as whether a flight was booked, a ticket was closed, or the code runs), and almost no evaluation measures whether the Agent noticed warnings in the context before executing.

The difficulty of evaluating Agents stems from non-deterministic outputs: the same input may produce different tool-calling paths and final answers, rendering the traditional "input–expected output" assertion logic ineffective.

The mainstream industry evaluation approach is to build a three-component set of "evaluation set + evaluation metrics + evaluation model," where the evaluation model is typically another strong model that scores automatically, supplemented by manual spot checks.

Evaluation is shifting from "scoring" to "diagnosis," not only telling developers how many points an agent scored but also pointing out which types of tasks and which stages it failed at.

For agents, evaluation needs to be continuous throughout the entire runtime rather than only before deployment, and building automated online evaluation mechanisms (evals) may become a standard operations practice.