12 Search APIs Tested: Any Two Random Providers Match the Best Single One on Hit Rate

Title

12 Search APIs Tested: Any Two Random Providers Match the Best Single One on Hit Rate

Meta Description

A Reddit benchmark shows: among 12 search APIs, any two random providers hit 68%, matching the best single one; stacking two raises it to 77%, while 16% of queries no provider can answer—the problem lies in agent search strategy, not vendor choice.

Article Content

When building AI agents, choosing a search API is often treated as a step requiring careful selection. But a set of benchmark data recently shared on the Reddit community delivers a counterintuitive conclusion: randomly picking two search APIs to form a fallback combination yields a hit rate almost identical to the best-performing single vendor.

Benchmark Setup and Core Data

According to the Reddit user, the test covered 12 search API vendors, used the same set of 363 queries, and evaluated whether the known answer appeared in the top 5 results. The results are as follows:

• 1 vendor only (best): 68% hit rate
• 2 vendors (stacked): 77% hit rate
• 3 vendors: about 80% hit rate
• All 12 vendors: 84% hit rate

The most surprising number: randomly selecting two vendors also reaches a 68% hit rate, on par with the best-performing single vendor. This means that, statistically, your carefully chosen "optimal solution" is no more reliable than simply pulling in two random vendors as a backstop.

Hit Rate is a commonly used evaluation metric in information retrieval, specifically referring to the proportion of returned result lists that contain a known correct answer. In this article, the "top 5 results" qualifier corresponds to the commonly used Recall@5 (recall at 5) concept in retrieval systems, which measures a system's ability to find the correct answer within a limited display window rather than ranking quality (such as MRR or NDCG). This metric is especially critical for search-augmented agents, because agents typically feed only the first few results into subsequent reasoning, and results further down are often truncated and discarded. It is worth noting that Recall@5 cannot reflect the specific position of the answer within the results, so two vendors with the same hit rate may still differ in performance within an actual agent pipeline due to different distributions of answer rankings.

Why Is "Having a Fallback" More Important Than "Choosing the Right Vendor"?

Looking at the data, the jump in hit rate from 68% to 77% occurs when going from 1 vendor to 2, after which marginal returns diminish rapidly—about 80% for 3 vendors and only 84% for 12. This suggests that search API error patterns have a degree of independence: when one vendor fails to find something, another vendor is likely to find it, and the probability of complementarity is far greater than the probability of repeated failure.

For agent developers, the practical implication of this conclusion is: rather than spending time grinding through benchmarks in search of the "one best" search API, it is better to quickly integrate two vendors as a fallback. The trigger mechanism can be switching when the first round misses, or routing by query type.

Fallback in distributed systems refers to a fault-tolerance pattern that automatically switches to a backup path when the primary path fails. In the search API context, the benefit of fallback depends on the correlation of error patterns across multiple vendors: if vendors fail on different queries due to different index sources, crawling strategies, and ranking algorithms, then failure events are approximately independent, and the combined hit rate after stacking approaches (1-(1-p)^n); if they share upstream data sources or systematically fail on similar queries, then complementarity is greatly reduced. In this article's data, the roughly 9 percentage point improvement in hit rate from 1 vendor to 2 is precisely a manifestation of low-correlation error patterns. In practice, one must also note that fallback introduces additional latency and consistency overhead, requiring trade-offs in timeout thresholds and retry strategies.

16% of Queries Cannot Be Found by Anyone, and Switching Vendors Cannot Fix It

There is another key number in the test: 16% of queries had no answer appear across all 12 vendors. This portion of failures is unrelated to vendor choice; the problem lies in the agent's own search strategy or page-reading capability.

The original author's interpretation is that this belongs to the level of "how to search" or "whether to read the page"—for example, poorly constructed queries, or the need to read deep into the page body to extract the answer while the API only returns summary snippets. In other words, the optimization direction is not changing vendors but improving the agent's retrieval logic.

Two Strategies: Fallback-on-Miss vs. Fan-Out Every Time

The original article raises an open question: should one use "fallback-on-miss" (switch after the first round fails) or "fanning out" (send requests to multiple vendors simultaneously for every query)?

The former saves call costs but has slightly higher latency; the latter responds faster but multiplies expenses. Combined with the data, since the second vendor provides the largest marginal contribution, fallback-on-miss already captures most of the benefit; fanning out is more suitable for scenarios that are extremely latency-sensitive and can afford query costs.

Fan-out refers to issuing requests to multiple backends in parallel for the same query and merging the results, as opposed to serial fallback. The core advantage of fan-out is that latency depends on the slowest rather than the sum, but the cost is a multiplied request volume and the need for result fusion logic (such as re-ranking by score and deduplication). Under this article's data, the second vendor provides the largest marginal hit contribution, so serial fallback-on-miss can obtain most of the benefit at lower cost; fan-out is better suited to scenarios that are extremely sensitive to first-byte latency and where vendor call costs are negligible. Another compromise is hedged request: after the first request is sent, if it does not return within a set time limit, send a second request in parallel, balancing latency and cost.

Implications for Agent Search Architecture

This set of data challenges the procurement mindset that "benchmarks decide everything." For teams building search-augmented agents, a more pragmatic approach is: first quickly launch a two-vendor fallback, then perform deep optimization such as query rewriting and page-level reading for those 16% stubborn failure cases. Vendor selection itself may not be as important as imagined.

One-line Summary

A benchmark of 12 search APIs shows that a random two-vendor fallback matches the hit rate of the best single vendor.

Paragraph Summary

A benchmark covering 12 search API vendors and 363 queries shows that using only the best single vendor yields a top-5 hit rate of 68%, while randomly selecting two vendors to form a fallback combination also reaches 68%. Stacking two vendors raises the hit rate to 77%, three vendors to about 80%, and all 12 vendors to only 84%, with marginal returns diminishing rapidly. This indicates that vendor error patterns have a degree of independence, and having a fallback is more important than carefully selecting a single vendor. Another 16% of queries have no answer across all vendors; the problem lies in the agent's search strategy or page-reading capability, not vendor choice. The author therefore recommends quickly launching a two-vendor fallback-on-miss first, then optimizing query rewriting and page-level reading for stubborn failure cases.