Agent Evaluation

When evaluating agents, blast radius should be the first dimension, depending on what the agent can do, the permissions it is granted, the environments it can reach, and whether errors can be rolled back quickly.

If MMO benchmarks become a trend, the dimensions of agent capability evaluation may need to be restructured, with planning efficiency, error recovery capability, resource management level, and interaction strategy all entering the scoring system, and task success rate no longer the only metric.

Rigorous agent research requires matching computational budgets, or at least reporting call counts and latency, to avoid computational confounding.

Large model evaluation, RAG evaluation and agent evaluation are three problems at different levels and cannot be handled with the same set of methodologies.

Evaluation is not an additional step after agent development is complete, but infrastructure that runs through the entire development process.

Agent evaluation must simultaneously satisfy three conditions—reasonable result, reasonable evidence, and a reasonable tool-calling chain—before an answer can be judged correct, and none of the three can be missing.

The instructor repeatedly emphasized in the project: no matter how many layers of evaluation are done, an agent can probably reach 99.9%, but it can never truly reach 100%.

Agent evaluation typically covers dimensions such as Faithfulness, Answer Relevancy, harmful content filtering, and Trajectory Correctness for multi-step tasks.

James Gung's research focuses on agent evaluation, dialogue simulation, and proactive agents.

Agent evaluation is widely recognized in the industry as a difficult problem, because the behavioral space of AI agents is complex, there may be multiple correct paths, and they are difficult to measure with a single metric.