Evals

The core problem with AI code generation is that models tend to produce tests that look reasonable rather than tests that capture real defects.

OpenAI将AGI定性为能在大多数经济价值任务上超越人类的AI系统

In multi-step agentic workflows, the final result is formed by a chain of intermediate decisions, and the same task may have multiple reasonable execution paths, making evaluation more difficult than traditional software testing.

Evals is a testing framework used to systematically measure the quality of model or agent outputs, analogous to unit tests and integration tests in software engineering.

A benchmark is a fixed set of questions used to score and horizontally compare models, similar to a mock exam, and is used to rank models.

AI testing introduces the concept of Evaluation, which typically uses continuous values from 0 to 1 for scoring, where evaluation is responsible for scoring and testing is responsible for setting the passing threshold.

AI product design requires introducing an evaluation system (Evals) to continuously measure the quality of model outputs, rather than relying on traditional functional testing.

OpenAI has publicly discussed that their model was specifically trained to efficiently use the apply_patch tool.

顶级AI公司通常设有专门的提示词工程团队，使用自动化评估框架（如OpenAI的Evals）对提示词进行量化评测

The Cursor Agent SDK has a built-in Evals feature that lets developers run a set of tests each time they modify instructions or add tools, to determine whether the agent's performance has improved or worsened.

Setting the number of evaluation runs is critical; a single pass does not prove anything, and multiple runs are needed to build confidence in the results.

In Evals, evaluation cases define the expected tool call sequence in JSON files, describing the behavior the Agent should exhibit.

Headhunter Mia stated that what is still in vogue is agents, especially Co-worker Agents oriented toward collaboration scenarios. Coding Agents represent a persistent talent gap, and talent related to Evals and pretraining data is also beginning to receive attention.

Once benchmark questions are made public they may be contaminated by training data, and a high score does not mean the model is equally reliable in a specific business scenario, so Evals are needed to supplement them.

Benchmarks answer how strong a model is, while Evals answer how an Agent performs in a specific scenario.

The lack of reliable Evals leads to an "optimization illusion" — where metric numbers look good but actual user experience has not improved or has even regressed

Building high-quality business Evals requires domain experts to design test sets, establish human-annotated golden sets, and track model iteration effects through continuous A/B comparisons

The difference between Evals and traditional software integration testing lies in the probabilistic nature of LLM outputs, which requires replacing exact string assertions with a golden question set plus expected-behavior judgments.

Because agent outputs are often open-ended text, Evals typically need to combine multiple approaches such as rule-based checks, comparison against human-annotated gold answers, and LLM-as-Judge scoring.

Error analysis involves systematically attributing root causes after Evals expose problems, including reasons such as insufficient retrieval recall, context window overflow, incorrect tool-call formatting, or the boundaries of the model's reasoning capability.