RAG Knowledge Conflict Classification: Can Decision Models Replace LLMs?

When a retrieval-augmented generation (RAG) system retrieves multiple sources that contradict each other, the generator needs to know the type of conflict in order to respond appropriately—whether there are multiple valid answers, the topic itself is disputed, the information is outdated, or one source is fabricated. Research by Cattan et al. (2025) shows that informing the generator of the conflict type improves answer quality by about 24 percentage points, but even frontier large language models (LLMs) struggle to accurately predict conflict types.

So does this classification task necessarily require an LLM? A recent paper explores the possibility of using typed decision models to identify knowledge conflict types. These models directly output probability distributions over a fixed set of options in a single forward pass rather than generating text, making them faster and cheaper.

Decision Models vs. Gemini: Weighing Accuracy and Cost

The research team evaluated all 11 decision models from 7 providers, served through a single commercial inference gateway, on the CONFLICTS benchmark using the original paper's prompts (verbatim), and compared them against the Gemini family on identical inputs.

When asked directly, decision models' accuracy ranged from 50.0%–67.0%. Ten of them matched or exceeded the Gemini 2.5 Flash used in the paper (61.3%), while costing 4–39× less (two were free) and with 2.6–6.0× lower latency. The current Gemini 3.8 Flash reaches 70.0% accuracy, significantly outperforming 8 decision models. But the researchers designed a cascading strategy: routing the most uncertain quarter of queries from each decision model to Gemini 3.8 Flash. As a result, 8 of the 11 models statistically matched pure Gemini 3.8 Flash at only 25–37% of its cost.

The significance of this finding: for a large number of "easy" conflict type judgments, small decision models are sufficient; only genuinely difficult edge cases require expensive large models. The cascading approach substantially reduces inference overhead while maintaining quality.

Typed decision models are a class of discriminative models that directly output probability distributions over a fixed label space. Unlike autoregressive LLMs that generate text, they compute logits for each candidate category in a single forward pass and apply softmax, so their output is naturally a probability vector over closed options rather than free text. These models are typically based on smaller encoder architectures (such as DeBERTa, ModernBERT, or small instruction-tuned models) and require no token-by-token decoding at inference, resulting in low latency and high throughput. The trade-off is the lack of an LLM's open-world knowledge and reasoning capability, limiting performance on difficult samples that require multi-step inference or external common sense. This is precisely the motivation for the cascading strategy described here: use decision models to cover high-confidence simple samples and route low-confidence samples to an LLM.

The CONFLICTS benchmark is a dataset built by Cattan et al. (2025) to evaluate knowledge conflict handling in RAG systems, covering multiple conflict types—multiple valid answers, topic disputability, outdated information, and fabricated sources. The benchmark requires models to determine the conflict category given retrieval sources and a query, and comes with standardized prompts and evaluation protocols. This paper adopts its original prompts (verbatim) to ensure comparability, allowing decision models and the Gemini family to be compared under identical inputs and the same label system, avoiding bias introduced by prompt differences.

End-to-End Performance: The Practical Utility of Type Labels

Classification accuracy is only an intermediate metric. The researchers further validated end-to-end effectiveness: when using the conflict types output by 10 decision models to guide generation, the frequency with which answers followed expected behavior was comparable to using LLM-generated types. This shows that the type labels provided by decision models are usable in the actual generation stage—not just a numbers game on a benchmark.

However, the study also revealed several limitations worth noting. After decomposing the classification scheme into a series of yes/no questions, all models' accuracy declined; fixes fitted on one model did not transfer to others. Pre-specified diagnostics showed that date information played no measurable role in classification, and models varied greatly in their sensitivity to source order. Additionally, decision models were less reliable than LLMs at detecting implanted fabricated sources as hard labels, but their output probabilities could still distinguish implanted sources from clean ones—meaning the probability information itself carries valuable signal.

Cascading (cascade routing) is an inference architecture that routes by confidence: first use a cheap model to process all requests, then based on the uncertainty of its output probabilities (such as maximum class probability, entropy, or margin), select the least reliable subset of queries to hand off to an expensive large model. This paper uses "the most uncertain quarter" as the routing threshold—i.e., the 25% of queries with the lowest top-1 probability from the decision model are sent to Gemini 3.8 Flash. The key to cascading is selective risk control—as long as high-confidence samples are classified accurately enough, overall quality can approach that of a pure large-model solution while average cost drops substantially. The trade-off is introducing a tunable hyperparameter (the routing ratio) that must be calibrated per task and cost budget; moreover, if the decision model's confidence is poorly calibrated, the routing decision itself becomes distorted, so calibration methods such as temperature scaling are often needed in practice.

Practical Implications: When Should You Use Decision Models?

This study provides clear guidance for engineering practice in RAG systems. For routine classification of knowledge conflict types, decision models offer significant advantages in cost and latency, making them especially suitable for high-throughput scenarios. The cascading architecture offers a pragmatic compromise: use decision models for most queries and let a large model handle the uncertain minority as a fallback.

But the study also cautions that how the classification scheme is decomposed significantly affects performance, and that fixes are not transferable across models—meaning deployment requires validation specific to the model and task. Fabricated source detection remains a weakness of decision models; in security-sensitive scenarios, hard-label judgments may still need to rely on LLMs.

The research team has released the code, prompts, and all model outputs on GitHub (https://github.com/Parv17k/decision-models-conflict-typing), providing a foundation for reproduction and extension.

One-line Summary

Small decision models can perform RAG knowledge conflict classification at low cost, and cascading with large models balances quality and expense.

Paragraph Summary

The article explores the feasibility of replacing LLMs with typed decision models for RAG knowledge conflict type classification. The study evaluated 11 decision models from 7 providers on the CONFLICTS benchmark; when asked directly, accuracy ranged from 50.0%–67.0%, with 10 models matching or exceeding Gemini 2.5 Flash's 61.3% while costing 4–39× less and with 2.6–6.0× lower latency. Against the stronger Gemini 3.8 Flash (70.0%), the researchers designed a cascading strategy: routing each decision model's most uncertain quarter of queries to the large model. As a result, 8 of 11 models statistically matched pure Gemini 3.8 Flash at only 25–37% of its cost. End-to-end validation showed that conflict types output by decision models guided generation as effectively as LLM-generated types. The article also notes several limitations: decomposing into yes/no questions reduces accuracy, fixes do not transfer across models, and hard-label fabricated source detection is less reliable than with LLMs—though probability outputs still carry useful signal.