QASecClaw Experiment: Reliability Boundaries of Multi-Agent Collaboration and Verifier Feedback

When deterministic tools can already provide evidence, can adding more context from specialized language models improve software assurance outcomes? A study titled Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance explores this question through five experiments covering Java warning control, OWASP Benchmark evaluation, Defects4J test generation, SV-COMP assertion determination, and deliberately misleading evidence packages. The experiments used Qwen2.5-Coder-7B-Instruct, supplemented by a 1.5B model for sensitivity analysis, completing a total of 13,638 model requests.

Experimental Design and Core Findings

The study is organized around three threads: "context quantity scaling," "verifier feedback," and "evidence reliability." On eight relevant development warnings, increasing the context quantity from 1 to 50 increased token usage by 49x but did not improve the rate of unsafe code elimination. This means that simply scaling up model context, when deterministic evidence already exists, yields very limited benefit and may even lead to wasted cost.

On 160 OWASP cases, different model configurations traded off between "cleared" and "unresolved" outcomes; the ten-context experimental group evaluated only 96 cases, exposing limitations in evaluation coverage. The study also found that on 32 SV-COMP tasks, raw verifier feedback raised the average number of correct model determinations from 12.7 to 22.0, but simultaneously increased incorrect determinations from 1 to 7. A JBMC-plus-JVM verification baseline returned 26 correct determinations and 6 unresolved results. This indicates that verifier feedback can improve accuracy but may also introduce new errors, requiring careful interpretation.

QASecClaw is an agent-based software assurance framework whose core idea is to combine language models with deterministic tools (such as static analyzers and verifiers), with agents coordinating evidence collection and reasoning. In this study, "Agent Scaling" refers to expanding system capabilities by increasing the number of agents or the scale of context, rather than simply enlarging model parameter counts. Ware et al. proposed the QASecClaw prototype in prior work, and this paper further systematically evaluates the effectiveness boundaries of scaling strategies.

The OWASP Benchmark is a Java vulnerability detection benchmark maintained by the OWASP Foundation, containing thousands of deliberately constructed vulnerability and false-positive cases, used to evaluate the detection capabilities of static analysis tools and AI models on real vulnerability patterns. Defects4J is a widely used Java defect dataset that provides defect versions and test cases from real projects, commonly used to evaluate automated program repair and test generation techniques. SV-COMP is an international software verification competition whose tasks cover assertion checking, reachability analysis, and more, providing a standardized benchmark for verifiers.

The Double-Edged Sword of Verifier Feedback

Verifier feedback appears to be an effective means of improving model determination quality, but the experimental data reveal its complexity. While the number of correct determinations rose significantly, incorrect determinations also multiplied. This means the model may over-rely on feedback signals, treating some ambiguous or erroneous feedback as reliable evidence, leading to misjudgments.

The JBMC-plus-JVM verification baseline compared in the study performed more robustly: more correct determinations and fewer unresolved results. This suggests that in software assurance scenarios, the way deterministic verification tools are combined with model reasoning needs careful design—not simply handing feedback to the model, but establishing an evidence admission mechanism to ensure only reliable evidence enters the model's decision-making process.

JBMC (Java Bounded Model Checker) is a bounded model checker for Java bytecode in the CBMC toolchain, capable of symbolic execution and assertion verification of Java programs. The JVM verification baseline refers to the traditional dynamic verification method of running programs directly on the JVM and checking assertions. The baseline formed by combining the two leverages JBMC's static exhaustive capability and JVM's dynamic execution results to corroborate each other, thereby achieving higher correct determination counts and fewer unresolved results on SV-COMP tasks. This combination reflects the advantage of deterministic tools in evidence generation: they do not rely on probabilistic reasoning, and their outputs are reproducible and auditable. In contrast, when language models adjust determinations based on feedback signals, they may internalize ambiguous or erroneous feedback as decision-making evidence, leading to increased misjudgments.

Evidence Reliability and Misleading Inputs

In tests with two sets of 120 evidence packages each, the model rejected 70% to 100% of applicable counterexamples, while checks constructed for specific scopes correctly classified all evidence packages. This result indicates that the model has some ability to recognize misleading evidence, but is not infallible; and rule-construction-based checks are more reliable within specific scopes.

On three Defects4J defects, no target defects were found, and one defect was not even supported by the test adapter. This further indicates that current evaluation coverage is limited, and the model's ability to discover real defects remains constrained by test infrastructure and evidence quality.

Implications for Software Assurance Practice

These descriptive results support adopting deterministic evidence admission strategies in test environments, while exposing limitations in determination aggregation, feedback interpretation, and evaluation coverage. For practitioners, blindly scaling up the number of agents or context size is not advisable; a more effective path is to build reliable evidence screening mechanisms, clarify the applicable boundaries of verifier feedback, and continuously expand evaluation coverage.

The study does not conclude that "agent scaling is a panacea"; rather, it reminds us that in high-risk domains such as software assurance, the reliability of deterministic tools remains the cornerstone, and language models are better suited as assistants rather than replacements. Future work needs to validate these findings on more real defects and richer evidence types.

One-line Summary

Through five experiments, the study shows that when deterministic evidence already exists, scaling language model context or agent size yields limited benefits; verifier feedback improves correct determinations but introduces more errors.

Paragraph Summary

The study is organized around three threads—"context quantity scaling," "verifier feedback," and "evidence reliability"—using models such as Qwen2.5-Coder-7B-Instruct to complete 13,638 requests. Experiments found: increasing context from 1 to 50 increased token usage by 49x but did not improve unsafe code elimination rates; raw verifier feedback raised average correct determinations from 12.7 to 22.0, while incorrect determinations increased from 1 to 7, and the JBMC+JVM baseline was more robust; the model rejected 70%-100% of applicable counterexamples, and constructed checks correctly classified all evidence packages; no target defects were found among the three Defects4J defects. The results support adopting deterministic evidence admission strategies in test environments and expose limitations in determination aggregation, feedback interpretation, and evaluation coverage.