The STRATA Benchmark: Making AI Clinical Reasoning Accountable Instead of Off-the-Cuff

When language models are asked "did sales drop after the product recall" or "is a certain side effect more common after vaccination," they often produce fluent sentences that read like conclusions—but with no statistical test behind them. Assertions about how events unfold over time are everywhere online, and each one is a temporal hypothesis that can only be settled by running a significance test on the right records.

The problem is that the verifier needs to specify the null hypothesis, choose a test method, and decide which records are relevant to the hypothesis. Any of these steps can go wrong, while the final conclusion may still happen to be correct. Existing language model data analysis benchmarks only judge the conclusion, and they hand the relevant data directly to the model. In health, wrong conclusions are costly. To address this, researchers propose STRATA—a benchmark that scores every decision on the path to a conclusion—along with the ART paradigm, which attempts to bridge the "right conclusion, wrong evidence" gap.

Benchmarks that only judge conclusions hide the real problem

The current mainstream approach to evaluating data analysis is to give the model prepared data and then check only whether its final conclusion is correct. This design ignores a key fact: in real settings, the model must first decide which records are relevant to the hypothesis, then select an appropriate statistical test, and only then reach a conclusion.

The research team built the STRATA benchmark in the health domain, containing 8,735 hypotheses about clinical timelines in vaccine adverse event reports. Each hypothesis comes with a gold-standard plan, evidence set, p-value, and conclusion. These annotations were generated under a public codebook and reproduced by a deterministic oracle. This means scoring no longer just checks "did it get the answer right," but examines the null hypothesis setup, test selection, evidence filtering, and final judgment item by item.

This fine-grained scoring reveals a phenomenon masked by accuracy: the system mainly fails at the step of "choosing which records count." In other words, the model may give the correct conclusion, but the evidence set supporting it is wrong—a phenomenon the researchers call the "conclusion-evidence gap."

Null hypothesis and significance testing: In statistical inference, the null hypothesis usually states "no effect" or "no difference," for example, "sales are unchanged before and after the recall." A significance test computes the probability of observing the current data (or more extreme data) if the null hypothesis were true—the p-value. When the p-value is below a preset threshold (often 0.05), the researcher rejects the null hypothesis and considers the effect significant. The key point is that the p-value itself does not measure effect size and does not prove causation; it only answers "how compatible the data are with the null hypothesis." Therefore, the same data can lead to opposite conclusions under different null hypotheses and different test methods. This is exactly why STRATA requires item-by-item checking of the null hypothesis setup and test selection—a correct conclusion may be mere luck, while a wrong testing path means the conclusion is not reproducible.

The ART paradigm: turning every decision into an inspectable output

To address these issues, the researchers propose the ART (articulate, retrieve, test) three-stage paradigm: first explicitly state the hypothesis and testing plan, then retrieve relevant records, and finally run the test. The decisions at each stage are exposed as inspectable outputs rather than hidden inside the model's internal reasoning.

Experiments across five model configurations show that ART closes most of the conclusion-evidence gap. A 31B open-source model under the same paradigm narrows the gap with frontier models to within 0.01; and frontier models can also successfully complete the task once given the codebook. This suggests that models do not lack statistical ability—they lack an explicit process for aligning hypotheses, evidence, and tests.

Conclusion-evidence gap: This refers to the inconsistency between the model's final judgment and the evidence set supporting that judgment. Specifically, the model may correctly answer "did sales decline," but the record filtering, time window, or test method it relies on does not support the conclusion, or even contradicts it. This gap is completely invisible in benchmarks that only judge conclusions, because accuracy compares only the final label. In health, the gap is especially dangerous: a correct conclusion built on wrong evidence may fail on a different dataset or population, and the source of the error cannot be traced. STRATA makes the gap quantifiable through item-by-item scoring, while ART compresses the space in which the gap can survive by forcing explicit outputs of hypothesis, evidence, and test.

What this means for AI medical applications

The value of STRATA goes beyond a single benchmark. It decomposes "significance testing of temporal hypotheses" into auditable steps, which has lessons for any AI system that needs to make causal or temporal inferences on real data. In clinical decision support, pharmacovigilance, public health surveillance, and similar settings, a conclusion that cannot be traced to a specific evidence set and test method is hard to trust.

The ART paradigm offers an engineering path: rather than expecting the model to produce the right answer in one shot, let it leave verifiable traces at every step. When the model is required to first articulate the hypothesis, then retrieve evidence, and finally test, errors are easier to detect and correct within the process.

The research also leaves open questions: should access to the codebook be part of model capability? Can the gap between open-source and frontier models still be closed on more complex hypotheses? These await future work. But at the very least, STRATA advances evaluation from the single criterion of "is the conclusion right" to the level of "is the reasoning process right."

Auditability: In high-risk settings such as clinical decision support and pharmacovigilance, auditability requires that every judgment of a system be traceable to specific inputs, methods, and intermediate decisions. Unlike "black-box accuracy," an auditable system allows reviewers to check whether the null hypothesis is reasonable, whether the evidence set is complete, and whether the test method is applicable, and to pinpoint the specific step when an error is found. STRATA decomposes the validation of temporal hypotheses into auditable steps, essentially building a "decision chain" for AI systems: the output of each step is the input to the next, and each can be independently reviewed. This is crucial for regulatory approval, retrospective attribution, and continuous monitoring—a conclusion that cannot be audited, even if correct, is hard to trust and adopt in medical practice.

One-line Summary

STRATA scores temporal hypothesis validation step by step, exposing the conclusion-evidence gap, and the ART paradigm bridges it.

Paragraph Summary

Language models often give fluent conclusions about temporal hypotheses without statistical testing. Existing benchmarks only judge conclusions and provide data directly, masking the real point of failure. Researchers propose the STRATA benchmark, containing 8,735 hypotheses about clinical timelines in vaccine adverse events, scoring the null hypothesis, test selection, evidence set, and conclusion item by item. They find that systems mainly fail at "choosing which records count," creating a conclusion-evidence gap. The accompanying ART paradigm splits reasoning into articulation, retrieval, and testing stages, making decisions inspectable. Experiments show that a 31B open-source model under ART narrows the gap with frontier models to within 0.01, and frontier models also succeed once given the codebook. STRATA advances evaluation from whether conclusions are right to whether the reasoning process is right, offering lessons for the auditability of AI medical applications.