Beyond Tag Search: Embedding + LLM Two-Stage Architecture Doubles Evidence Recall

Pinpointing every relevant piece of evidence across a vast literature is a shared challenge for systematic reviews, evidence-based medicine, and even enterprise knowledge bases. The traditional approach is to tag each document with terms from a controlled vocabulary and search by tag. Tags organize results neatly, but they miss items that don't match any tag, and broad tags drag in a flood of noise. A study on a large corpus of randomized trials offers a new answer: use text embeddings to semantically recall candidates, then use a language model to filter out the truly relevant evidence. This two-stage architecture raises the recall of evidence needed for systematic reviews from 45% to 92%.

Where Tag-Based Search Hits Its Ceiling

The study compared two approaches on a corpus of 7,354 randomized trials. Tag-based search relies on a manually maintained controlled vocabulary. Its strengths are tidy, highly interpretable results, but the price is incomplete coverage: if a piece of evidence cannot be accurately described by existing tags, it is nearly impossible to recall. More troublesome is the difficulty of calibrating tag granularity—too fine and a large number of items end up with no usable tag; too broad and every search returns overwhelming results.

The numbers from the study are striking: for the same set of evidence that a systematic review needed to include, tag-based search found only 45%. That means more than half of the relevant evidence was missed at the very first step. For systematic reviews that demand "not a single one left behind," such recall is clearly insufficient.

Controlled vocabulary is a manual indexing tool in information retrieval, in which domain experts predefine a set of standardized terms and establish hierarchical, equivalence, and associative relationships among them. Common controlled vocabularies include MeSH (Medical Subject Headings) in medicine and the Ei Thesaurus in engineering. Indexers assign subject terms to each document based on the vocabulary, and users query using standardized terms from the vocabulary. The advantage of controlled vocabularies is that they eliminate ambiguity and synonymy in natural language, making search results highly consistent; their inherent limitation is that coverage is bounded by the size of the vocabulary—emerging concepts or cross-disciplinary expressions are often not captured by existing vocabularies, leaving relevant documents unindexed and unrecallable.

The Two-Stage Design: Embedding Recall Plus LLM Filtering

The new method splits retrieval into two steps. The first step uses text embeddings to find candidates semantically: both the query and the documents are mapped into a vector space, and similarity is used to recall a set of potentially relevant items. This step depends on no predefined tags, so evidence is never missed simply because "no tag matches."

The second step hands query-specific evidence filtering to a language model: for each candidate item, it judges whether it truly satisfies the current query's needs and keeps only the relevant ones. This preserves the high coverage of semantic recall while using model judgment to suppress the noise from broad recall. The study summarizes this as a two-stage architecture of "broad candidate recall + query-specific evidence filtering," and notes that the cost per query rises with retrieval depth—the deeper the recall, the more candidates the model must judge, and the costs naturally grow.

Text embedding is a technique that maps a piece of text into a high-dimensional dense vector, such that semantically similar texts are also close together in vector space. Typical implementations include Transformer-based encoders (such as Sentence-BERT, E5, and BGE), trained via contrastive learning on large-scale corpora so that the model learns to encode queries and documents into the same semantic space. At retrieval time, the system computes cosine similarity between the query vector and document vectors and recalls candidates ranked by similarity. Embedding-based retrieval does not depend on any manual tags, so it can cover expressions outside the tagging system; however, its similarity is continuous and global, and it cannot precisely determine whether an item satisfies the fine-grained conditions of a specific query—which is exactly why the second-stage model filtering is needed.

The Trade-offs Behind 92% Recall

From 45% to 92%—the improvement in recall is an order of magnitude. For tasks like systematic reviews and evidence synthesis, where missing a single item can affect the conclusions, this difference is decisive. But the study also honestly presents the cost: it is not fixed, but grows with search depth. In other words, pursuing higher recall requires paying model invocation costs for more candidates.

This suggests a practical engineering judgment: the value of the two-stage architecture is not in unconditionally replacing tag-based search, but in turning "coverage" and "cost" into adjustable knobs. For scenarios with extremely high recall requirements, the candidate pool can be deepened; for latency- and cost-sensitive scenarios, the candidate range can be narrowed. Tag-based search remains suitable for organizing results and coarse filtering, but for the goal of "finding all the evidence," semantic recall plus model filtering shows a clear advantage.

Recall is a core evaluation metric in information retrieval, defined as the proportion of relevant items found by the retrieval system out of all relevant items in the corpus. In systematic review settings, the full set of relevant items is typically determined by manual screening and serves as the evaluation baseline. Recall and precision form a tension: expanding the candidate pool can raise recall, but it introduces more irrelevant items, lowers precision, and increases the burden on downstream human or model screening. In this article, the 45% recall of tag-based search means more than half of the relevant evidence never entered the candidate set—an omission that in evidence synthesis could directly change the direction of conclusions, which is why recall takes priority over precision in this setting.

Implications for AI Retrieval Systems

Many AI systems face the same structural problem: first find all relevant items in a large set, then organize what was found. This study shows that assigning "find everything" and "filter accurately" to two different mechanisms is more effective than using a single set of tags to do both jobs. Embeddings excel at broad coverage; language models excel at fine-grained judgment based on query intent—the combination is complementary.

Of course, this approach has points that warrant further observation: how to guarantee the judgment quality of the language model filtering, whether the cost curve rising with depth is acceptable, and whether the same recall gains can be reproduced on corpora from different domains. But at least for the task of randomized-trial evidence retrieval, the data already points in a clear direction.

(This article is based on a research summary from an rss source and introduces no data beyond the summary.)

Background

Systematic review is a rigorous research method in evidence-based medicine that, for a specific clinical question, systematically searches, screens, appraises, and synthesizes all relevant primary studies. Its methodology requires pre-registering the search strategy and including evidence as comprehensively as possible to reduce selection bias. The evidence retrieval stage of a systematic review typically involves screening thousands to tens of thousands of records, traditionally relying on combined searches of controlled vocabulary and free-text terms, followed by manual title and abstract screening. Incomplete searching is one of the main sources of bias in systematic reviews, so improving recall and reducing missed detections is a long-standing methodological concern in the field—and the core pain point that the two-stage approach in this article targets.