Can LLM Reasoning Beat KNN Voting? The Boundaries of Retrieval-Augmented Tabular Classification

Retrieval-augmented in-context learning (ICL) allows large language models (LLMs) to infer labels for tabular data from similar examples. But is this "LLM reasoning" actually more effective than simply performing weighted voting over the retrieved neighbors? A new paper provides a nuanced answer through a rigorous comparison of 4 general-purpose LLMs and distance-weighted KNN (Distance-KNN) across 19 tabular classification datasets: LLMs do not always have the upper hand—their value depends heavily on the quality and distance of the retrieved neighborhood.

Experimental Setup: A Fair Comparison Between LLMs and KNN

The research team designed a controlled-variable experiment: for each test sample, they first used the same retriever to obtain a set of neighbor samples, then fed these exact same neighbors as demonstrations to both the LLM and Distance-KNN. This way, any performance difference can only be attributed to the "reasoning method" itself, not to retrieval quality.

The comparison included four general-purpose LLMs (unnamed but mainstream models) and a distance-weighted nearest-neighbor voter. Evaluation covered 19 tabular classification datasets, systematically varying the size of the retrieval candidate pool, the number of demonstrations, prompt formats, retrieval representations, and label-balanced demonstration selection strategies.

Distance-KNN (Distance-Weighted KNN) is a classic instance-based learning algorithm. In tabular classification tasks, it first finds the K nearest neighbors to a query sample in the feature space using some distance metric (e.g., Euclidean distance or cosine similarity), then votes based on the neighbors' labels. Unlike simple majority voting, the distance-weighted version assigns each neighbor a weight, typically inversely proportional to distance (e.g., weight = 1 / (distance + ε)), so that closer neighbors have a greater influence on the final prediction. This method is simple, requires no training, and has a clear statistical interpretation, often serving as a strong baseline for tabular data classification. In the context of this paper, Distance-KNN uses exactly the same retrieved neighbors as the LLM, ensuring a fair comparison.

Key Finding: KNN Consistently Leads in Average Accuracy

With the number of demonstrations fixed, expanding the retrieval candidate pool improves accuracy for all predictors—this is intuitive, as a larger pool means a higher chance of finding truly relevant neighbors. However, at every pool size setting, Distance-KNN achieves the highest average accuracy.

In other words, on global average, LLM reasoning does not outperform simple distance-weighted voting. This result directly challenges the "LLMs are universal" narrative and reminds researchers that in tabular ICL, one cannot assume LLM reasoning will necessarily bring gains.

Retrieval pool refers to the set of candidate samples initially retrieved by the system during similarity search. In retrieval-augmented in-context learning (ICL), a large number of candidates (e.g., 50 or 100) are typically retrieved from the entire training set first, and then a small number of samples are selected as demonstrations (e.g., Top-K) according to some strategy and fed to the LLM. The size of the candidate pool directly affects retrieval quality: a larger pool increases the probability of including relevant neighbors but may also introduce more noise. By systematically varying pool size (e.g., from 10 to 100), this paper observes that accuracy for all predictors improves with pool size, reflecting the fundamental impact of retrieval recall on final performance.

When Do LLMs Pull Ahead? Low Label Consensus and Close Neighborhoods

Deeper analysis in the paper reveals the conditions under which LLMs truly create value:

• Low label consensus: When the retrieved neighbors have inconsistent labels (i.e., neighbors "disagree"), LLM gains are most concentrated. In such cases, voting mechanisms are easily disrupted by noise, while LLMs may leverage pre-trained knowledge to make more reasonable inferences.
• Feature space distance: Under low label consensus, some LLMs can outperform Distance-KNN in "close" and "medium-distance" neighborhoods, but in the farthest distance group, all LLMs are on average behind.
• Model differences: Performance varies significantly across different LLMs, indicating that reasoning abilities are not homogeneous.

These findings delineate a clear boundary: LLM reasoning advantages are not universal but local and conditional.

Label consensus refers to the degree of agreement among retrieved neighbor samples on class labels. It is typically measured by the proportion of the majority class among neighbors or by entropy. High consensus means neighbor labels are nearly identical, in which case simple voting yields high-confidence predictions; low consensus means neighbor labels are dispersed, and voting results are susceptible to noise. In low-consensus scenarios, LLMs may leverage their vast pre-trained knowledge to infer more reasonable decision boundaries from demonstrations, thereby surpassing mechanical voting. This paper finds that LLM gains are concentrated here, suggesting that their reasoning ability is more valuable when information is ambiguous.

Robustness Checks: Are the Conclusions Stable?

The research team further tested the number of demonstrations, prompt phrasing, retrieval representations (e.g., different embedding methods), and label-balanced demonstration selection. These supplementary experiments "largely preserve" the main conclusions: KNN's average advantage is robust, and LLM gains remain concentrated in low-consensus, close-distance scenarios.

This means the findings are not accidental results of a particular configuration but reflect a structural regularity in retrieval-based tabular ICL.

Practical Implications

For engineers building tabular classification systems, this study conveys several pragmatic signals:

1. Don't assume LLM reasoning is better by default: When retrieval quality is assured and neighbor labels are consistent, Distance-KNN is a simpler and on-average more accurate baseline.
2. Focus on disagreement samples: When neighbor labels are messy, LLMs may be worth introducing as a complementary signal beyond voting.
3. Distance is an important moderating variable: For queries far from the training distribution, LLMs can't save the day either; better retrieval or rejection strategies may be needed.
4. Choose models carefully: Reasoning gains in tabular ICL vary significantly across LLMs and cannot be generalized.

This study does not deny the potential of LLMs in tabular tasks but precisely delineates their "value-added interval" relative to classical retrieval voting. At a time when retrieval-augmented generation (RAG) and tabular data are increasingly intertwined, this boundary awareness is more valuable than blindly stacking models.

Rejection option refers to a classification system choosing not to make a prediction when encountering low-confidence samples or those far from the training distribution, instead handing them to human processing or marking them as unknown. In tabular classification, when a query sample is far from all retrieved neighbors, neither KNN nor LLM can predict reliably. Introducing a rejection mechanism in such cases can avoid high-risk errors and improve overall system reliability. This paper notes that LLMs on average lag in the farthest distance group, implying that relying solely on LLM reasoning cannot compensate for insufficient retrieval quality; a rejection strategy incorporating distance thresholds is a more pragmatic choice.

One-line Summary

Across 19 tabular classification datasets, distance-weighted KNN achieves higher average accuracy than LLM reasoning; LLMs only pull ahead under low label consensus and close distances.

Paragraph Summary

This study compares four general-purpose LLMs with distance-weighted KNN (Distance-KNN) on 19 tabular classification datasets through controlled-variable experiments. Both use identical retrieved neighbors, varying only the reasoning method. Results show that at every retrieval pool size, Distance-KNN's average accuracy exceeds that of LLMs. However, LLMs can outperform KNN when retrieved neighbors have low label consensus and are close in feature space, while all LLMs on average lag in the farthest distance group. Moreover, performance varies significantly across LLMs. The main conclusions remain robust after varying the number of demonstrations, prompt formats, retrieval representations, and label-balancing strategies. Practical implications: when retrieval quality is high and labels are consistent, KNN is a stronger baseline; LLMs are suitable as a complementary signal for disagreement samples; queries far from the training distribution require better retrieval or rejection strategies.