Can Non-Generative Model JEV Meet Clinical Trial Standard Evaluation Needs?

Introduction: Pain Points in Clinical Trial Pre-Screening

Clinical trial pre-screening is a high-risk, high-cost process. Researchers need to determine whether individuals meet enrollment criteria from massive patient records, while also distinguishing between "conditions not met" and "insufficient evidence"—two fundamentally different situations. Traditionally, this process relies on manual review, which is inefficient and prone to missing information. In recent years, large language models (LLMs) have been introduced into this scenario, but their generative nature may bring hallucinations and instability, especially in clinical environments requiring strict privacy protection.

A new study compared the non-generative decision model JEV with schema-constrained Qwen Flash on clinical trial criteria evaluation tasks. JEV directly returns probabilities for predefined options rather than generating free text, giving it a natural advantage in structured decision-making tasks. Results vary across datasets but overall reveal a significant advantage of non-generative models in recalling "insufficient information" conditions.

Study Design: Rigorous Comparison Under ZDR

Data and Tasks

The study drew 2,048 pairs of test patient-condition pairs from TrialGPT and Q&A datasets built on MIMIC-III and MIMIC-IV. Each sample includes patient records, standard questions, and predefined answer options. Models must provide a judgment for each condition: met, not met, or insufficient information. All evaluations were conducted under strict Zero Data Retention (ZDR) policies to ensure clinical data privacy.

Zero Data Retention (ZDR) is a strict data privacy protection policy requiring service providers to immediately delete all user data after processing requests—no storage, no logging, no use for model training or any secondary purposes. In clinical environments, ZDR is particularly critical because patient records contain highly sensitive personal health information subject to regulations like HIPAA and GDPR. Adopting ZDR means model inference must be completed in stateless, log-free pipelines, which typically imposes additional requirements on latency and cost and limits solutions requiring persistent context or cache optimization. This study evaluated models under ZDR, ensuring the transferability of conclusions to real-world privacy-compliant scenarios.

Comparison Models

• JEV: A non-generative decision model that directly outputs probability distributions over native options.
• Qwen Flash: A schema-constrained generative model whose answers are restricted to predefined options.

Both models evaluated conditions in single and batch modes, with batch mode sharing the same patient context. Performance was measured against reference labels using macro-averaged F1 scores (macro-F1).

Schema-constrained generation refers to forcing generative models to select answers only from predefined enumerated values through structured output specifications (such as JSON Schema or function calling interfaces) rather than freely generating text. This approach compresses the generative model's output space to a finite set of options, significantly reducing hallucination risk and improving answer parsability. However, constraints only affect output format—the model internally still generates token by token in an autoregressive manner and may hesitate among options due to probability distribution shifts, especially tending to "force a choice" rather than acknowledging insufficient information when evidence is ambiguous. This fundamentally differs from non-generative models that directly output native option probabilities.

Key Findings: JEV Shows Clear Advantages in Insufficient Information Scenarios

Performance Comparison in Batch Mode

In batch mode, JEV and Qwen showed similar macro-F1 scores on TrialGPT and MIMIC-IV:

Dataset	JEV macro-F1	Qwen macro-F1
TrialGPT	0.652	0.660
MIMIC-IV	0.807	0.814
MIMIC-III	0.768	0.534

On MIMIC-III, JEV significantly outperformed Qwen (Δ = +0.234, 95% CI: 0.187-0.281). This advantage mainly stems from JEV's significantly higher recall for "insufficient information" conditions: 56.3% vs. 9.9%. This means JEV is better at identifying conditions that cannot be determined from existing records, thereby prompting human review and avoiding erroneous exclusion.

Macro-averaged F1 (macro-F1) is a metric that first calculates the harmonic mean (F1) of precision and recall for each category separately, then takes the arithmetic mean across all categories. Unlike micro-averaging, macro-averaging gives equal weight to each category, making it extremely sensitive to recall performance on minority classes (such as "insufficient information"). In clinical trial pre-screening, "insufficient information" often has low prevalence but significant clinical importance—missing it can lead to erroneous patient exclusion. Qwen's sharp drop in macro-F1 on MIMIC-III occurs precisely because it almost never predicts the "insufficient information" class (recall only 9.9%), causing that class's F1 to approach zero and dragging down the overall average. Understanding this metric characteristic is essential to correctly interpret the source of JEV's advantage.

Latency and Cost

Across all three datasets, JEV showed lower observed median task latency (1.18-1.40 seconds vs. 2.80-8.90 seconds) and reduced per-condition estimated cost by 53%-74%. For large-scale pre-screening scenarios requiring processing of numerous patient-condition pairs, this advantage has practical significance.

Interpretation: JEV's Positioning and Limitations

As a Preliminary Assessment Component

The study positions JEV as a "preliminary assessment component": it provides criterion-level judgments and flags potential information gaps for human review. Its non-generative nature makes outputs stable, interpretable, and free of hallucinated text, which is particularly important in clinical decision support.

Errors Remain, Requiring Human Verification

Although JEV excels at recalling insufficient information conditions, the study also observed errors in its interpretation of clinical evidence. The authors emphasize that before using JEV's outputs for eligibility decisions, they must be verified against patient records. This reminds us that any automated tool in clinical settings should serve as an aid rather than a replacement.

Conclusion: Value and Boundaries of Non-Generative Models

This study offers a new approach for clinical trial pre-screening: non-generative decision models can achieve comparable or even better performance than generative models in structured option tasks at lower cost and latency, especially in identifying insufficient information conditions. However, their error patterns also indicate that clinical evidence interpretation still requires caution. In the future, combining models like JEV with human review processes may achieve a balance between efficiency and safety.

For practitioners focused on AI implementation in healthcare, this comparison also provides an important perspective: not all tasks require generative large models—non-generative models designed for specific decision scenarios may offer advantages in reliability, cost, and speed.

One-line Summary

Non-generative decision model JEV significantly outperforms generative Qwen in identifying insufficient information conditions in clinical trial pre-screening, with lower latency and cost.

Paragraph Summary

A study compared non-generative decision model JEV with schema-constrained generative model Qwen Flash on clinical trial criteria evaluation tasks under Zero Data Retention (ZDR) policies. Based on 2,048 patient-condition pairs from TrialGPT and MIMIC datasets, JEV achieved macro-F1 of 0.768 on MIMIC-III, far ahead of Qwen's 0.534, mainly due to high recall for "insufficient information" conditions (56.3% vs 9.9%). Performance was similar on TrialGPT and MIMIC-IV. JEV's median latency was 1.18-1.40 seconds versus Qwen's 2.80-8.90 seconds, with per-condition cost reduced by 53%-74%. The study positions JEV as a preliminary assessment component that can flag information gaps for human review, but its clinical evidence interpretation errors still require verification against patient records.