Pre-Registered Test Fails: 4B Model Replacing 12B, Fault-Injection Regression Gate Proves My Own Prediction Wrong

Pre-registered experiment disproves author's prediction: small model replacing large fails fault-injection regression gate.
ARCI v2.0 adds a fault-injection regression gate for LangChain's create_agent, comparing baseline vs. candidate configs on the same task set with tool timeouts, errors, and empty results injected. The author pre-registered an experiment predicting a 4B model could replace a 12B model, but it was BLOCKed: 12B passed 96/96, 4B passed 70/96, and 23 of 26 failures were the same behavior (passing a nonexistent product code). The small model also sometimes skipped query steps so faults never triggered, which would overestimate its robustness if not counted separately. The tool currently refuses async Agents and other scenarios, with conclusions from a single run only.
Pre-Registered Test Fails: 4B Model Replacing 12B, Fault-Injection Regression Gate Proves My Own Prediction Wrong

A maintainer added a LangChain adapter to his Agent regression tool ARCI, putting create_agent through a fault-injecting regression gate. He pre-registered the comparison plan and wrote down his prediction before running — and the prediction lost. The most valuable part of this post isn't "small models don't work," but rather that he used a reproducible experimental structure to turn the most gut-feel part of Agent model selection into a question that can be adjudicated as PASS / BLOCK in advance.
What ARCI v2.0 Does: Wrapping create_agent in a Fault-Injection Regression Gate
ARCI is an open-source regression gate for Agents. The core addition in v2.0 is the LangChain adapter: you hand it your tools and create_agent settings, and it runs the same task set against both a baseline and a candidate configuration, while injecting three types of faults: tool timeouts, tool errors, and empty results.
The key design principle is "define the rules first, then run the experiment": the task set and decision rules must be fixed before the run, and the output is only one of four verdicts — PASS, BLOCK, INCONCLUSIVE, or ERROR. There's no room for after-the-fact tuning, and no ambiguous "let's run it again and see" zone.
The integration cost is also low. The example adapter file is about 50 lines: define your tools, write two settings functions, call make_adapter once, and you're connected. For teams already building Agents with LangChain, the barrier to entry is close to "just add a CI check while you're at it."
Fault Injection is a method of proactively testing system robustness by artificially introducing controlled errors. In Agent scenarios, common fault types include: tool call timeouts (simulating slow downstream service responses), tools returning errors (simulating API exceptions), and tools returning empty results (simulating missing data). Traditional unit tests only verify the "happy path," whereas fault injection forces the model to operate under non-ideal conditions, exposing its ability to perceive and recover from anomalies. ARCI combines fault injection with a regression gate, meaning every candidate configuration change must pass the same fault-laden task set, preventing models from performing well in "fair weather" but collapsing at the first sign of trouble.
Pre-registration originates from scientific research and refers to publicly registering hypotheses, methods, and decision criteria before an experiment begins, to avoid cherry-picking favorable results after the fact. ARCI engineers this principle: the task set and decision rules must be locked before the run, and only four verdicts are output afterward — PASS/BLOCK/INCONCLUSIVE/ERROR — eliminating any cheating room for "adjusting thresholds after the run." This practice is especially important in model selection because it transforms subjective debate into reproducible objective adjudication; even if the result contradicts the author's expectations, it's still trustworthy.
The Pre-Registered Bet: Can 4B Replace 12B?
Before running, the author wrote down his prediction: replacing a 12B model with a 4B local model on the 24-task "order-desk" test suite, he believed it would PASS.
The result was BLOCK.
- 12B model: 96 out of 96 trials passed.
- 4B model: 70 out of 96 trials passed.
- Actual injected fault trigger counts by fault condition: 12B was 24/24, 4B was 18 to 20/24.
The entire experiment ran on a laptop with Ollama, took 18 minutes, and required no API key. This cost structure is itself part of the conclusion: running an Agent regression gate as a daily process doesn't require heavy infrastructure.
23 of 26 Failures Were the Same Behavior
The concentration of failure modes far exceeded expectations. Of the 26 failures, 23 were the same error: the small model, when calling the inventory tool, passed in a product code that didn't exist for that order. This wasn't a diffuse "slightly weaker capability" gap — it was a very specific, individually targetable failure point.
The engineering takeaway is direct: if you only look at overall pass rate, you'll conclude "the small model is a notch worse"; but if you cluster by failure behavior, you get a concrete item you can write into a prompt, a tool schema validation, or a unit test. The value of a regression gate is precisely in surfacing these actionable details rather than giving a vague score.
Second Finding: The Small Model Cut Corners, So Faults Never Triggered
The author specifically emphasized this as the part he didn't anticipate: the small model sometimes skipped the query step entirely, so the injected fault never had a chance to trigger.
This creates a hidden statistical trap — if you don't separately count whether faults actually triggered, the comparison under fault conditions looks like a fair contest. In reality, some "passes" may be because the model bypassed the tested path rather than correctly handling the fault. This is a universal warning for anyone doing fault-injection testing: the injection action itself needs to be verified as executed, otherwise test results will systematically overestimate the system under test.
Injection Trigger Verification is an easily overlooked step in fault-injection testing. When the test framework injects a fault (e.g., making a tool time out), it must independently verify whether that fault was actually executed. If the model under test simply skips the tool call where the fault was injected, the fault becomes moot, and the model may pass the test by "bypassing" it. ARCI separately counted actual fault trigger occurrences and found that the 4B model triggered faults only 18-20 times out of 24 fault conditions, while the 12B triggered all of them. Without separate counting, the pass rate under fault conditions would systematically overestimate the small model's robustness, because some passes stem from never entering the test path. For any team doing fault-injection regression, "whether the fault was activated" should be output as an independent metric — otherwise comparison results will be distorted.
Known Boundaries: What It Explicitly Refuses Today
ARCI currently loudly errors out rather than silently passing for several scenarios: async Agents, interrupts, tools with injected parameters, and tool artifacts. The author's exact words were "loudly rather than silently" — better to refuse than to pretend it ran.
The author also proactively stated the limitations of the results: conclusions apply only to this registered test suite, and only to a single run of 96 paired trials. A single run means no statistical significance can be derived — it's only a directionally clear signal.
To reproduce it yourself:
pip install "agent-reliability-ci[langchain]"
arci init --langchain
arci run arci.yaml
The repository and pre-registration documents are under the agent-reliability-ci project on GitHub, and the results file is phase0/plan/exec/v2/study/RESULT.md.
Why This Matters
Agent model selection has long relied on demos, gut feel, and leaderboards. This experiment offers a different paradigm: lock down the decision rules first, then compare, turning "can we switch to a smaller model?" from a debate into a CI step that outputs PASS or BLOCK. The fact that the author's own prediction was overturned actually proves the credibility of this structure — if pre-registered experiments always confirmed the author's intuition, they'd mostly just be decoration for intuition.
One-line Summary
Pre-registered experiment disproves author's prediction: small model replacing large model fails fault-injection regression gate.
Paragraph Summary
ARCI v2.0 adds a fault-injection regression gate for LangChain's create_agent, allowing baseline vs. candidate configurations to be compared on the same task set with three fault types injected: tool timeouts, errors, and empty results. The author pre-registered an experiment predicting a 4B model could replace a 12B model, but it was BLOCKed: 12B passed 96/96, 4B passed 70/96, and 23 of 26 failures were the same behavior (passing a nonexistent product code). Additionally, the small model sometimes skipped query steps so faults never triggered, which would overestimate its robustness if not counted separately. The tool currently explicitly refuses async Agents and other scenarios, and conclusions apply only to a single run.
Related articles

AI Could Overthrow Governments in Six Months? ARC Head Issues His Starkest Warning Yet
ARC head Paul Christiano warns AI capable of overthrowing governments may arrive in 6-12 months, with ~one-third odds of superintelligence within a year. He urges frontier labs to pause algorithmic efficiency gains and fix safety first, and reveals how the Hugging Face incident foreshadowed AI taking over evaluation.

How Do Spies Infiltrate Nvidia? A CIA Agent Reveals a Honeypot Operation
A former CIA agent reveals in a YouTube interview that intelligence agencies ran a honeypot seduction operation against Nvidia CEO Jensen Huang, exposing the hidden spy risks facing tech executives.

How NVIDIA Cosmos Helps Delivery Robots Rehearse Unexpected Situations in Advance
NVIDIA Cosmos World Foundation Model expands a small amount of real input into large-scale synthetic scenarios, enabling delivery robots to train for unexpected situations in simulation in advance. Cocoa Robotics uses it to cover unexpected events and combines Isaac Sim and Isaac Lab to evaluate perception, prediction, and path planning.