LLM-IRA: Quantum-Inspired Interactive Ranking Aggregation for Boosting RAG Retrieval with Active LLM Feedback

Retrieval-augmented generation (RAG) systems depend heavily on the quality of their underlying retrievers, and since a single retriever often has blind spots, fusing multiple retrievers is a practical way to improve recall and ranking. But how to assign reasonable fusion weights to different retrievers has long been an underappreciated problem. A recent paper proposes LLM-IRA, which extends the quantum-inspired interactive ranking aggregation framework to document reranking for the first time, offering a new approach to LLM-driven retrieval fusion.

Core Idea: Query the LLM Only on the "Most Disputed" Document Pairs

The key design of LLM-IRA is to "spend effort where it matters most." Rather than asking the LLM to judge every document pair, it actively selects the document pairs where multiple retrievers disagree most, has the LLM provide preference judgments, and then learns the fusion weights of each retriever from those.

This interactive learning approach has two advantages. First, the LLM only makes pairwise judgments—a simple task with stable output—avoiding the cost and uncertainty of lengthy listwise prompts. Second, the final ranking is produced entirely by a single linear score fusion, which is extremely fast and requires neither maintaining a full pairwise matrix nor calling the LLM online to rerank items one by one.

Pairwise vs. listwise judgments: A pairwise judgment only requires the LLM to compare the relative merits of two documents (A better than B, B better than A, or a tie), which has a small output space, short prompts, and stable results. A listwise judgment requires the LLM to give a complete ranking or scores for a set of documents all at once, which involves longer prompts, more prone to position bias and inconsistency in the output, and costs that rise with the number of candidates. LLM-IRA chooses the former precisely to minimize the LLM's reasoning burden while making the preference signals easy to aggregate into weights.

Linear score fusion: A common approach to multi-retriever fusion is to multiply each retriever's normalized score by a weight and sum them to obtain the final ranking score. Its advantage is that the inference stage requires only a single weighted sum, with complexity linear in the number of documents, no need to store a pairwise comparison matrix between documents, and no reliance on online LLM calls. The quality of fusion hinges on whether the weights are reasonable, and LLM-IRA's contribution is precisely to estimate these weights using a small number of LLM preference judgments.

Two Weight Update Strategies

The paper reports two weight update variants: momentum blending and full accumulated-history refit. The authors conduct experiments on the BEIR benchmark and the TREC Deep Learning tracks, using the open-weight instruction-tuned model Llama-3.1-8B-Instruct as the judge. Results show that LLM-IRA matches or numerically exceeds strong static fusion baselines on 5 of the 6 evaluation benchmarks.

In terms of statistical significance, the two variants combined achieve a significant advantage with Wilcoxon signed-rank test p<0.05 over the best unsupervised aggregator on 4 of the 6 benchmarks: DL19 is significant under momentum blending, while DL20, DBpedia, and NFCorpus are significant under full accumulated-history refit. The authors note that the latter is overall the stronger configuration.

Momentum blending vs. full accumulated-history refit: Momentum blending is an online update method in which the LLM preferences obtained each round are blended into the current weights only in a certain proportion, with old information gradually decayed. It is therefore more sensitive to recent disputed samples and computationally cheap, but may forget early signals. Full accumulated-history refit retains all collected preference pairs and re-solves the aggregation weights each round, equivalent to a globally optimal fit using ever-growing data; it is usually more stable and accurate, at the cost of per-round computation that grows with the history size. The two represent a trade-off between "fast and light" and "stable and accurate," which also explains why the authors consider the latter stronger overall.

Wilcoxon signed-rank test: This is a non-parametric statistical test used to determine whether the difference between two sets of paired results systematically deviates from zero. In information retrieval evaluation, it is often used to compare whether the metric differences between two systems on the same batch of queries are significant; p<0.05 means the observed advantage is unlikely to be caused by random fluctuation. Because it does not assume the metrics follow a normal distribution, it is suitable for bounded ranking metrics such as MRR and nDCG.

Fewer Calls, Lower Cost, and Approaching Zero-Shot Rerankers

Efficiency is the most noteworthy aspect of LLM-IRA. On the DL19 and DL20 benchmarks analyzed by the authors, the method already surpasses the best unsupervised baseline with only 5 LLM calls per query, far from exhausting the typical 25–30 call budget. This means it can converge quickly in the early stage, and calls can stop before marginal returns diminish.

Compared with state-of-the-art zero-shot LLM rerankers, LLM-IRA achieves comparable cost-effectiveness with an order of magnitude fewer LLM calls. At the same time, the entire method requires neither training data nor model fine-tuning, making it a plug-and-play solution.

Zero-shot LLM rerankers: These methods directly have the LLM read the query and candidate documents and output relevance judgments or rankings, requiring no task-specific training. They are highly general but costly—each query often requires calling the LLM on dozens of candidates individually or in groups, and the input length and output uncertainty of listwise prompts both drive up overhead. LLM-IRA has a different goal: it does not have the LLM directly produce the final ranking, but treats the LLM as a "preference annotator" to estimate fusion weights, so the number of calls can be compressed to single digits while the heavy lifting of reranking is left to linear fusion. This also explains why it can achieve comparable cost-effectiveness with an order of magnitude fewer calls.

Implications for RAG System Deployment

From an engineering perspective, the value of LLM-IRA lies in turning "multi-retriever fusion" from an empirical task of manually tuning weights into a process supported by a theoretical framework and executable interactively. The quantum-inspired framework was previously mostly used in interactive ranking aggregation scenarios; this extension to document reranking shows that the framework can connect with modern retrieval pipelines.

Of course, the method has its boundaries. The experiments mainly revolve around the judge model Llama-3.1-8B-Instruct, and performance with a different judge model is not yet clear; whether the strategy of actively selecting disputed samples remains efficient when the number of retrievers increases and the disagreement structure becomes more complex also needs further verification. In addition, the significant benchmarks in the paper are concentrated on specific datasets, and cross-domain generalization remains to be evaluated more thoroughly.

For teams building RAG pipelines, LLM-IRA offers a simple but practical judgment: there is no need to involve the LLM in every reranking step—just let it "vote" at the genuinely informative points of disagreement, and you can gain a visible improvement in ranking quality at a very small calling cost.

One-line Summary

LLM-IRA learns multi-retriever fusion weights from a small number of LLM preference judgments, improving RAG ranking quality at low cost.

Paragraph Summary

LLM-IRA extends the quantum-inspired interactive ranking aggregation framework to document reranking for the first time. Its core is to select only the document pairs where multiple retrievers disagree most, have the LLM make simple pairwise preference judgments, and then learn linear fusion weights for each retriever from those. The method offers two update strategies—momentum blending and full accumulated-history refit—and on 5 of 6 benchmarks across BEIR and TREC DL it matches or exceeds strong static fusion baselines, with 4 achieving a significant advantage under the Wilcoxon test at p<0.05. In terms of efficiency, it surpasses the best unsupervised baseline with only 5 LLM calls per query, far below the typical 25–30 call budget, and approaches the cost-effectiveness of zero-shot LLM rerankers with an order of magnitude fewer calls. The method requires no training data or fine-tuning and is plug-and-play, but judge model changes, increased retrievers, and cross-domain generalization remain to be verified.