LEANN Open-Source Vector Index: Compressing a 60M-Chunk RAG Index from 201GB to 6GB

Introduction

Retrieval-augmented generation (RAG) has become an important paradigm for deploying large models, but the storage overhead of vector indexes is often underestimated. LEANN, an open-source project from the UC Berkeley team, uses a "pruned graph + recompute at query time" approach to compress a 60M-chunk RAG index from 201 GB to 6 GB, drawing attention from the community.

Core Mechanism: Store the Graph, Not the Vectors

Traditional vector indexes (such as HNSW) need to persist a complete high-dimensional embedding vector for each data chunk, plus the graph structure. LEANN's approach is to store only the pruned nearest-neighbor graph, then recompute embeddings in real time based on candidate nodes in the graph at query time.

In other words, LEANN converts "storage-intensive" into "compute-intensive"—sacrificing some query latency in exchange for orders-of-magnitude lower storage usage. According to MarkTechPost, the index achieves accuracy comparable to HNSW at 90% recall, while memory usage is 4 GB versus 188 GB.

HNSW (Hierarchical Navigable Small World) is one of the mainstream approximate nearest neighbor search index structures today. It builds a multi-layer navigable small-world graph and searches from the coarse-grained upper layers downward layer by layer at query time, achieving logarithmic retrieval complexity. HNSW's advantage lies in high recall and low latency, but the cost is that each vector node must persist a complete high-dimensional embedding (for example, a 768-dimensional or 1536-dimensional float array), and the graph structure itself also needs to store adjacency relationships. For tens of millions of chunks, embedding vectors often account for the vast majority of the total index size—exactly the storage bottleneck that LEANN tries to circumvent.

Performance Trade-offs: Seconds-Level Search in Exchange for Space

The cost of this design is that query time rises to the seconds level. For offline batch processing, low-frequency retrieval, or storage-cost-sensitive scenarios, this is an acceptable trade-off; but for high-concurrency, low-latency online services, seconds-level latency remains a barrier.

Interestingly, recomputing embeddings means that queries need access to the original text or the embedding model, which introduces additional I/O and compute overhead. The team has clearly optimized this path in the paper, but the specific engineering details still need to be evaluated in conjunction with the paper and code.

Recall is a core metric for measuring the quality of approximate nearest neighbor search, usually defined as the proportion of the returned Top-K results that truly belong to the ground-truth nearest neighbor set. 90% recall means that about 1 in every 10 queries will miss some true nearest neighbors. For RAG scenarios, recall directly affects whether the retrieved context contains the information needed to answer the question, so 90% is generally regarded as an acceptable baseline. LEANN achieves accuracy comparable to HNSW at 90% recall, indicating that its strategy of pruned graph plus query-time recomputation does not significantly sacrifice quality, but this metric needs to be independently verified on different datasets.

Dataset Scale and Storage Ledger

Two data points from the original article are worth comparing: the 60M-chunk index is reduced from 201 GB to 6 GB; at 90% recall, memory usage drops from 188 GB to 4 GB. The former corresponds to disk index size, and the latter corresponds to memory residency at query time. The two have different definitions but both point to the same conclusion—LEANN achieves orders-of-magnitude compression in the storage dimension.

For RAG systems, index size directly determines deployment cost and scaling elasticity. When the index drops from hundreds of GB to single-digit GB, deploying a larger-scale knowledge base on a single machine becomes possible, and the burden of cold starts and backup migration also drops significantly.

Disk index size and query-time memory residency are two different measurement dimensions. Disk index size refers to the space occupied by index files on persistent storage, determining deployment cost, backup migration burden, and cold-start loading time; memory residency refers to the size of the data structures that need to be loaded into RAM during queries, directly affecting the number of concurrent queries a single machine can handle and hardware selection. LEANN compresses the disk index from 201 GB to 6 GB while reducing query memory from 188 GB to 4 GB, meaning it achieves not only compression in the storage dimension but also an order-of-magnitude reduction in query-time memory usage, which is especially critical for resource-constrained environments.

Applicable Scenarios and Limitations

LEANN is better suited to the following scenarios: large knowledge bases with low query frequency, internal retrieval that is not latency-sensitive, resource-constrained edge or single-machine deployments, and experimental environments that require frequent index rebuilding.

The limitations are also obvious: seconds-level retrieval is not suitable for the stringent latency requirements of interactive question answering; recomputing embeddings depends on the availability of original data; and the recall performance of the pruned graph needs to be verified for different data distributions. Whether to adopt it depends on the specific trade-off between latency and cost in the business.

Industry Significance

LEANN's value is not in "fully replacing HNSW," but in pointing out an overlooked optimization path: vector indexes do not have to treat embeddings as immutable assets to be stored, and the boundary between computation and storage can be redrawn. As the inference cost of embedding models continues to decline, the cost-effectiveness of "recompute at query time" will further improve, and this type of design is expected to secure a place in RAG infrastructure.

For developers, LEANN is an open-source project, so they can directly evaluate its code and paper, verify recall and latency performance on their own datasets, rather than making decisions based solely on promotional data.

Embedding model inference cost is a key variable in the economics of the "recompute at query time" approach. In recent years, the inference efficiency of embedding models (such as Transformer-based dual-tower encoders) has continued to improve, and the encoding cost per token has continued to decline, making it feasible to compute embeddings for candidate chunks in real time during the query phase. However, recomputing embeddings still requires access to the original text or precomputed features, which introduces additional I/O overhead; if the original text is stored in remote object storage, network latency may become a new bottleneck. Therefore, the actual cost-effectiveness of LEANN-style designs depends on the comprehensive trade-off among embedding model inference speed, original data access latency, and storage cost, rather than a single factor.

One-line Summary

LEANN, an open-source vector index, compresses a 60M-chunk RAG index from 201GB to 6GB by storing only a pruned graph and recomputing embeddings at query time.

Paragraph Summary

The LEANN project, open-sourced by the UC Berkeley team, proposes a vector index approach that "stores the graph, not the vectors": it persists only the pruned nearest-neighbor graph and recomputes candidate node embeddings in real time at query time. On a 60-million-chunk RAG index, LEANN compresses the disk index from 201GB to 6GB, and at 90% recall reduces query memory from 188GB to 4GB, with accuracy comparable to HNSW. The cost is that query latency rises to the seconds level, and recomputing embeddings requires access to original data or the embedding model, bringing additional I/O and compute overhead. This solution is suitable for large-scale low-frequency retrieval, resource-constrained deployments, and experimental environments, but not for low-latency online services. Its industry significance lies in pointing out an optimization path in which the boundary between computation and storage can be redrawn.