Hillock v1.0: Cracking Vector Retrieval's Hallucination Problem with Deterministic Search

Vector search is the mainstream approach for current RAG (Retrieval-Augmented Generation) systems, but it has a widely overlooked flaw: probabilistic retrieval produces hallucinations on "unanswerable queries." After 6 months of testing, the open-source project Hillock v1.0 has officially been released, attempting to solve this problem at the retrieval layer.

The Hallucination Root of Vector Search

The core logic of vector search is "nearest neighbor": encode both the query and documents as vectors, then find the closest document fragments. The problem is that this mechanism always returns results—even when the knowledge base contains nothing relevant.

According to the observations of the original Reddit poster, when users ask questions that the knowledge base cannot answer, vector retrieval still returns fragments that "seem relevant" semantically. After the large model receives these fragments, it often forcibly fabricates an answer. This is not a problem with the model itself, but rather the retrieval layer disguising "irrelevant" as "relevant."

In other words, vector search lacks a clear "refuse to answer" signal. Its probabilistic nature means similarity scores can only reflect semantic proximity, not whether the content truly supports an answer.

Nearest Neighbor Search is the foundational operation of vector retrieval: given a query vector, find the k vectors in the vector store with the smallest distance (commonly cosine similarity or Euclidean distance). Its mathematical properties dictate that as long as the store is non-empty, there will always be a "nearest" neighbor—the system has no built-in "no results" branch. In practice, systems typically also set similarity thresholds to filter low-quality results, but thresholds are globally fixed and difficult to adapt to the semantic needs of different queries—too high a threshold misses genuinely relevant long-tail content, while too low a threshold lets through large amounts of weakly relevant fragments, which is precisely the direct channel through which "pseudo-relevance" is fed to the generation side.

Hillock v1.0's Approach

Hillock's positioning is not to replace vector databases, but to introduce deterministic judgment at the retrieval layer. According to the author, v1.0 went through 6 months of testing iterations, with the core goal of enabling the system to clearly identify and refuse unanswerable queries rather than returning plausible-sounding fragments.

This approach aligns with recent industry discussions: the reliability bottleneck of RAG systems is often not on the generation side but on the retrieval side. If the retrieval results themselves are unreliable, even the strongest model can only "reason correctly based on incorrect information."

Deterministic Retrieval vs Probabilistic Retrieval

Traditional vector retrieval outputs a ranked list, with each result carrying a similarity score. Hillock attempts to add a layer of judgment beyond similarity: do these results actually constitute valid support for the query?

This "determinism" does not mean completely eliminating probability, but rather making "unable to answer" a legitimate output. For scenarios such as enterprise knowledge bases and customer service bots, explicitly saying "I don't know" is far more valuable than fabricating an answer.

RAG (Retrieval-Augmented Generation) typically works as follows: encode the user query as a vector, retrieve Top-K document fragments from the knowledge base, concatenate them into the prompt, and hand them to the large model to generate an answer. This architecture externalizes "knowledge" to the retrieval layer, so the quality of retrieval results directly determines the credibility of generated content. The industry saying "garbage in, garbage out" is especially evident in RAG: the generative model cannot distinguish whether a fragment truly supports an answer or is merely semantically similar—it tends to treat context as factual evidence. This is why the retrieval layer's "ability to refuse" is regarded as a key element of RAG reliability.

Similarity scores (such as cosine similarity, typically ranging from -1 to 1) measure the directional proximity of query vectors and document vectors in the embedding space. They reflect semantic relevance but do not equal "this fragment can answer the question." Two texts may discuss the same topic yet state different facts, and their similarity can still be high. Therefore, score ranking alone cannot distinguish "relevant" from "supportive," which is why Hillock advocates adding a layer of judgment beyond similarity: treating "whether it constitutes valid support" as an independently decidable question, rather than relying on score thresholds.

Implications for RAG Developers

The release of Hillock v1.0 reminds us to re-examine the design assumptions of the retrieval layer. Most RAG tutorials assume "retrieval must return Top-K," but in real-world business scenarios, a large number of queries inherently fall outside the knowledge base's scope.

When evaluating retrieval solutions, developers can test "the ability to handle unanswerable queries" as an independent metric rather than only looking at recall. A system that performs well on standard QA sets may not remain honest on edge-case queries.

Hillock is currently released as open source; for specific implementation details and benchmark data, it is recommended to consult the project repository directly. For teams building production-grade RAG systems, tools targeting retrieval-layer flaws like this are worth including in technical selection comparisons.

Conclusion

Vector search is not going away, but its probabilistic flaws need to be confronted. Hillock v1.0 offers a noteworthy approach: teaching retrieval systems to say "I don't know" may be the most direct step toward reducing RAG hallucinations.