DeepEval

Evaluation dimensions for the tool-calling layer include whether the Agent selects the correct tool, whether it passes correctly formatted and semantically reasonable parameters, and whether it can correctly parse the results returned by the tool

Testing tool-calling accuracy requires attention to three levels: tool selection accuracy, parameter extraction accuracy, and tool chain orchestration rationality, of which parameter extraction is often the most error-prone.

Common open-source evaluation solutions include LangSmith, Braintrust, DeepEval and others, which provide tracing, dataset management and metric computation capabilities.

The study's generation-side evaluation used DeepEval's claim-level metrics, with deepseek-v4-pro serving as the LLM judge, supplemented by traditional BLEU/ROUGE metrics.

Common tool-calling evaluation frameworks break tool failures into two major categories: "wrong tool selected" and "correct tool selected but wrong parameters," with parameter errors further divided into structure and value validation.

DeepEval releases Tool Correctness and Argument Correctness as two independent metrics.

Automated evaluation tools such as RAGAS or DeepEval can be used to define KPIs for each step of an Agent, replacing judging quality by eye.

When choosing only one framework, if you rely heavily on pytest and have no YAML-driven evaluation system, you should choose DeepEval, but you must fix evaluation_steps on day one.

After cleaning up the anchors and prompts, promptfoo's agreement rate rose from 34/40 to 37/40, and DeepEval's rose from 35/40 to 36/40.

promptfoo integrates with CI via CLI and YAML, while DeepEval integrates via pytest.

DeepEval's first-time configuration took about 40 minutes, running 40 data items took 4 minutes 48 seconds, the cost was $1.31, and the agreement rate with manual annotation was 31/40.

DeepEval does not raise an error when the sum of scoring rubric weights is not 1.0, and performs no sanity check.

When evaluation_steps is not passed, DeepEval's G-Eval metric automatically generates scoring steps based on criteria, causing the same response to potentially get opposite conclusions on re-runs, producing drift.

In the article, under DeepEval's default mode, 7 responses had their pass/fail conclusion flip at least once across three re-runs; after manually fixing evaluation_steps, the number of flips dropped to 3, and the agreement rate rose from 31/40 to 35/40.

RAGAS and DeepEval are commonly used agent/model evaluation frameworks.

Evaluation frameworks such as Ragas and DeepEval focus on offline batch testing of RAG pipeline or single-inference quality, making them suitable for running in CI workflows.

The AI agent evaluation (LLM Evaluation) space has seen the emergence of representative products such as Ragas, DeepEval, LangSmith, and Brainlid.

LLM evaluation engineering typically relies on frameworks such as RAGAS and DeepEval, combined with human annotation or using another LLM as a judge for scoring.

Mainstream RAG evaluation frameworks include RAGAS, TruLens, and DeepEval, which typically score across two dimensions—retrieval quality and generation quality—but lack built-in support for specialized requirements such as exact identifier matching and rule clause completeness

Ragas、TruLens、DeepEval等开源框架提供RAG指标的自动化计算，其中不少采用LLM as a judge方式评分