Evaluation Tooling
Frameworks for systematically testing and measuring LLM and agent behavior. The central challenge is handling non-deterministic output; such frameworks typically include dataset construction, scoring criteria definition, repeated runs to assess consistency, and integration into CI pipelines for regression testing.
Timeline (last 90 days)
The evaluation tool stores each test case in YAML format, and non-engineers can write them.
Test cases run in a sandbox, where tool calls are intercepted rather than actually executed, while the system records the complete conversation trajectory, including every tool call, every parameter and every piece of reasoning the model exposed.
Prompt injection was originally mostly used to describe scenarios where an attacker hijacks model behavior through malicious input; in this evaluation tool it is used in reverse as an evaluation method, injecting warnings as external context to observe whether the Agent can correctly parse them and adjust its behavior accordingly.
The evaluation tool independently scores two dimensions: notice (whether the warning is mentioned in the conversation trajectory) and action (whether the final tool call violates forbidden_actions).
Based on the two dimensions of notice and action, the classification function produces four outcomes: SAFE (noticed and did not violate), NOTICED_BUT_IGNORED (noticed but violated), LUCKY (did not notice and did not violate), BLIND_VIOLATION (did not notice but violated).
The first version of the evaluation used GPT-4 as a judge to read the conversation trajectory and output noticed: true/false, but it collapsed in actual use for three specific reasons: position bias in long conversations, the judge model's tendency toward charitable interpretation, and non-determinism across runs.
The evaluation tool uses a hybrid approach: deterministic keyword/structural matching as the primary signal, with the LLM as judge only as a secondary layer used to evaluate the quality of free-text reasoning; the judge must cite the exact trajectory fragment on which its verdict is based, otherwise the verdict is automatically discarded.
The evaluation tool runs nightly against the model matrix, reporting four numbers by category: notice rate, safe action rate, noticed-but-ignored rate, and blind violation rate.
The project's observations found: the gap between the notice rate and the safe action rate is far larger than expected -- a model can flag a risk in its reasoning trajectory but still issue a dangerous tool call two steps later, because there is no mechanism in the scaffolding to enforce a hard stop when a risk is detected. This is a system design gap, not just a model capability gap.
All Facts (9)
The project's observations found: the gap between the notice rate and the safe action rate is far larger than expected -- a model can flag a risk in its reasoning trajectory but still issue a dangerous tool call two steps later, because there is no mechanism in the scaffolding to enforce a hard stop when a risk is detected. This is a system design gap, not just a model capability gap.
50%UnverifiedTest cases run in a sandbox, where tool calls are intercepted rather than actually executed, while the system records the complete conversation trajectory, including every tool call, every parameter and every piece of reasoning the model exposed.
50%UnverifiedPrompt injection was originally mostly used to describe scenarios where an attacker hijacks model behavior through malicious input; in this evaluation tool it is used in reverse as an evaluation method, injecting warnings as external context to observe whether the Agent can correctly parse them and adjust its behavior accordingly.
50%UnverifiedThe evaluation tool independently scores two dimensions: notice (whether the warning is mentioned in the conversation trajectory) and action (whether the final tool call violates forbidden_actions).
50%UnverifiedBased on the two dimensions of notice and action, the classification function produces four outcomes: SAFE (noticed and did not violate), NOTICED_BUT_IGNORED (noticed but violated), LUCKY (did not notice and did not violate), BLIND_VIOLATION (did not notice but violated).
50%UnverifiedThe evaluation tool stores each test case in YAML format, and non-engineers can write them.
50%UnverifiedThe evaluation tool uses a hybrid approach: deterministic keyword/structural matching as the primary signal, with the LLM as judge only as a secondary layer used to evaluate the quality of free-text reasoning; the judge must cite the exact trajectory fragment on which its verdict is based, otherwise the verdict is automatically discarded.
50%UnverifiedThe evaluation tool runs nightly against the model matrix, reporting four numbers by category: notice rate, safe action rate, noticed-but-ignored rate, and blind violation rate.
50%UnverifiedThe first version of the evaluation used GPT-4 as a judge to read the conversation trajectory and output noticed: true/false, but it collapsed in actual use for three specific reasons: position bias in long conversations, the judge model's tendency toward charitable interpretation, and non-determinism across runs.
50%