Multi-Step AI Agent Guardrail in Practice: Fact-Change Detection Drives False-Alarm Rate to 0%

The Problem: Every Step Correct, Yet the Sequence Wrong

AI Agent reliability issues often don't lie in a single tool call, but in multi-step sequences. Each step, viewed in isolation, is compliant and reasonable—but when combined, several steps drift from user intent, or even trigger irreversible operations. A developer shared on Reddit's r/AI_Agents community the problem he raised a few weeks ago: how do you stop an Agent from making this kind of "multi-step error"? Now he's delivered the first runnable version of a solution.

The key insight behind this approach: before an irreversible action executes, re-read the facts it depends on, and if the relevant facts have changed since approval, block it. Sounds simple, but the devil is in the word "relevant"—if every change is treated as a blocking signal, false alarms will overwhelm the system.

The Guardrail Mechanism: Only Check the Facts an Action Actually Depends On

The guardrail's core logic is "pre-execution fact re-verification." When an irreversible action is about to run, the system re-reads the set of facts that action depends on and compares them against the snapshot taken at approval time. If any relevant fact has changed, the action is blocked.

The difficulty lies in defining "relevant." The author compared strategy differences across 23 test scenarios on a benchmark of two tools:

• Block on any change: Block as soon as any fact changes—false-alarm rate as high as 50%.
• Dependency-fact check + field boundaries: Only check the facts an action truly depends on, and set boundaries for fields allowed to vary within a limited range—false-alarm rate drops to 17%.
• Adding conservative normalization: Without lowering the catch rate, the false-alarm rate drops further to 0%.

This progression illustrates an important fact: the precision of a guardrail doesn't depend on detection capability, but on the granularity with which "what counts as a change" is defined. Coarse-grained full comparison is safe but nearly unusable in real systems; fine-grained dependency tracking is the viable path.

The system runs on the local model qwen2.5-7b, meaning the solution doesn't depend on cloud APIs, keeping deployment costs and data-privacy concerns low.

Facts and Fact Snapshots

In the guardrail context, a "fact" refers to atomic information that an action's execution depends on and that can be read from system state—such as account balance, order status, permission flags, or file version numbers. A fact snapshot is a one-time, read-only record of these facts at the moment of approval. During pre-execution re-verification, the system compares current facts against the snapshot item by item rather than re-evaluating the entire system state. This design confines blocking logic to the declared dependency set of an action, avoids false alarms triggered by irrelevant state fluctuations, and turns the definition of "relevant change" from vague semantics into an enumerable field list.

Conservative Normalization

Normalization refers to standardizing fact values before comparison—for example, removing timestamp precision differences, unifying case, or ignoring known-harmless formatting variations. Conservative normalization emphasizes performing only standardizations that won't mask real changes; its goal is to eliminate false alarms caused by "surface changes" without reducing the catch rate for substantive changes. The author relies on this step to bring the false-alarm rate from 17% down to 0%, showing that false alarms often stem from overly fine comparison granularity rather than a flaw in the detection logic itself.

Benchmark Design: Catch Rate and False-Alarm Rate Must Be Reported Separately

The author specifically emphasizes that the benchmark reports two independent numbers: catch rate and false-alarm rate. This design choice is worth noting—a guardrail evaluation that reports only catch rate is incomplete, because a strategy that "blocks everything" has a 100% catch rate but no practical value.

Only by reporting both metrics separately can the guardrail's ability to balance "blocking what should be blocked" and "letting through what should be let through" be truly reflected. The downward path of false-alarm rate from 50% to 17% to 0% is also meaningful only within a dual-metric framework.

Known Gaps: Disclosure-by-Combination and Lost-Ack Retry

The author openly admits that two scenario types previously raised by the community are listed as "known gaps" in the benchmark, which the current mechanism cannot catch:

• Disclosure-by-combination: A single piece of information disclosure is harmless, but multiple pieces combined constitute a sensitive information leak.
• Lost-ack-retry: Retries caused by lost acknowledgment signals leading to duplicate execution problems.

Writing gaps explicitly into the benchmark rather than covering them up is a rare, honest practice in this kind of engineering sharing, and it also sets boundaries for future iterations.

Mechanistic Explanation of Disclosure-by-Combination and Lost-Ack Retry

Disclosure-by-combination means a single information disclosure is not sensitive in isolation, but multiple disclosures combined can allow inference of sensitive conclusions. The guardrail checks dependency facts action by action and cannot model the cumulative reasoning effects across actions, so this type of risk falls into the blind spot of single-step checks. Lost-ack-retry means an action has already executed but the acknowledgment signal is lost in transit; the system misjudges it as a failure and retries, causing duplicate execution. The guardrail focuses on whether facts have changed since approval, but in retry scenarios the facts may not have changed—the problem lies in the reliability of the acknowledgment channel rather than fact state, so the current mechanism cannot catch it. The two gap types correspond respectively to cross-action combination and channel reliability, which are inherent boundaries of the dependency-fact re-verification paradigm.

Two Open Questions: Real False-Alarm Cases and Rule Writability

The author poses two questions to the community, both pointing to practical bottlenecks in deployment.

First, he needs real-world cases where "the fact changed, it looks like it should be blocked, but letting it through would actually be fine." Such cases are hard to construct out of thin air, yet they directly determine performance on the false-alarm side—designing normalization rules requires real data support.

Second, can the line between "changes that should be blocked" and "harmless changes" be hard-coded tool by tool? Or does it ultimately still require human judgment case by case? The answer to this question determines whether the guardrail can be configured at scale or degrades into an auxiliary tool for manual approval.

From an engineering perspective, the value of this guardrail version isn't that it has solved multi-step Agent safety, but that it decomposes the problem into two measurable, iterable dimensions: catching and false alarms. The benchmark scale of 23 scenarios is not large, and the 0% false-alarm conclusion needs validation on more real scenarios—but the path of "dependency-fact re-verification + field boundaries + conservative normalization" provides a reproducible starting point for protecting against irreversible operations in multi-step Agents.

For teams building Agent systems, this practice suggests a more general principle: a guardrail's effectiveness depends on precise modeling of "what an action depends on," not on broad monitoring of "what has changed."

One-line Summary

Developer proposes pre-execution fact re-verification guardrail, cutting multi-step Agent false-alarm rate from 50% to 0%.

Paragraph Summary

A developer shared on Reddit the first runnable version of a multi-step AI Agent guardrail. The core idea: before an irreversible action executes, re-read the facts it depends on and compare them against the approval-time snapshot, blocking if relevant changes are found. The difficulty is defining "relevant": across 23 test scenarios, blocking on any change yielded a 50% false-alarm rate, dependency-fact checks with field boundaries reduced it to 17%, and conservative normalization brought it to 0% without lowering the catch rate. The benchmark reports catch rate and false-alarm rate as two independent metrics. The author lists disclosure-by-combination and lost-ack-retry as known gaps, and poses two open questions to the community: real false-alarm cases, and whether rules can be hard-coded per tool. The system runs on the local qwen2.5-7b model.