CacheGraft: Injecting Large-Model Knowledge into Small-Model RAG via KV Cache

Retrieval-augmented generation (RAG) conventionally works by converting documents into text chunks and stuffing them into the prompt, but a new paper proposes an alternative path: encode documents offline into a fused KV cache between a large model and a small model, then inject that cache directly into the small model at query time—no further calls to the large model required. This is CacheGraft, built on the Cache-to-Cache (C2C) fusion framework, which attempts to replace text-level knowledge transfer with "cache grafting" in latent space.

From Agent Communication to RAG: An Unexpected Use for the C2C Framework

The C2C framework was originally motivated by letting transformer-based language model agents communicate not only through text but also directly through key-value (KV) cache state. The authors of CacheGraft argue that this "cache-to-cache" translation capability can be lifted out of its original real-time agent-communication setting and repurposed for retrieval-augmented generation.

Concretely: in an offline stage, each passage in the corpus is encoded into a fused KV cache representation between a large model (4B parameters in the experiments) and a small model (0.6B parameters). At query time, this cache can be injected directly into the small model without the large model being present. In other words, the large model's knowledge is "precompiled" into the cache, and the small model merely consumes it.

KV cache refers to the key (Key) and value (Value) tensors that a Transformer decoder caches layer by layer during autoregressive generation. For each new token generated, the model only needs to compute attention for the current token, reusing the K/V of historical tokens directly, thereby reducing generation complexity from quadratic to linear. The KV cache is essentially an "internal state snapshot" of the context the model has processed, with dimensions of layers × attention heads × sequence length × head dimension. Precisely because it carries contextual semantics and is decoupled from model weights, it can potentially be extracted, transformed, and injected into another model—which is exactly what C2C and CacheGraft operate on.

Fused KV cache refers to a cache that does not come from a single model, but rather a hybrid cache obtained by aligning, weighting, or transforming the KV representations produced by the large model with the small model's own KV representations through a fuser. Since different models may differ in hidden dimensions, layer counts, and attention head configurations, the fuser needs to learn cross-model projection mappings so that the large model's cache can be "translated" into a format the small model can directly consume. The key to CacheGraft is that the fuser completes this mapping in the offline stage, so at online time the small model can use the fused cache without the large model's involvement.

Zero-Shot Gains: Winning 29 out of 30 Comparisons

The paper evaluates on five question-answering benchmarks and uses two independently running local judge models. The results show that CacheGraft outperforms the pure-text receiver baseline in 29 of 30 comparisons, 17 of which are statistically significant.

More notably, these gains are zero-shot for the fuser—the fuser was never trained on retrieval data. This means the method's success is not due to task-specific tuning for RAG, but rather the generalization capability of the C2C framework itself.

Zero-shot here specifically means that the fuser has not undergone any training or fine-tuning on retrieval question-answering data; its parameters come entirely from the C2C framework's original training objective. During evaluation, the offline-generated fused cache is injected directly into the small model to answer questions, which constitutes an out-of-distribution generalization test. The 17 "statistically significant" results typically mean that across multiple samplings or runs, the method's improvement over the baseline exceeds random fluctuation (e.g., p<0.05), rather than judging superiority from a single result. The role of two independent local judge models is to reduce the preference bias of a single judge, but the judges themselves are also models, so their judgments may still introduce systematic bias—one reason the authors position their conclusions as "preliminary results."

Why Latent-Space Enrichment Is Attractive

One bottleneck of traditional RAG is that text chunks have limited information density, and small models often make inefficient use of long contexts. CacheGraft's idea is to compress document information into the higher-dimensional representation space of the KV cache, bypassing the text bottleneck.

This is especially meaningful for compute-constrained scenarios: if only a small model can be deployed at inference time but you want knowledge coverage approaching that of a large model, then generating caches offline with a large model and running only the small model online is an attractive compromise. The authors also explicitly note that latent-space document enrichment may be a promising direction for improving RAG system performance, particularly in compute-constrained environments where only small models can be deployed.

Latent space refers to the model's internal high-dimensional continuous representation space, as opposed to the discrete text token space. Traditional RAG cuts documents into text chunks and splices them into the prompt, with information limited by token count and surface form; latent-space enrichment encodes documents as continuous KV tensors, which in theory can carry denser semantics in the same or fewer "positions." The cost is reduced interpretability—knowledge in the cache is difficult to inspect, cite, or edit the way text chunks are. In addition, KV cache size grows with layer count and head dimension, so how to store, retrieve, and select relevant caches as corpus scale expands is a key engineering problem on the path to practicality for this approach.

Preliminary Results and Open Questions

The paper positions itself as "preliminary results." While the zero-shot performance is striking, the fact that the fuser was not trained on retrieval data also means there is room for further improvement through training. Cache injection compatibility, transfer effectiveness across different model pairs, and the scalability of cache representations as corpus size grows are all questions that must be answered before this method becomes practical.

From a broader perspective, CacheGraft points to a less-explored direction: RAG enhancement need not happen only at the text layer—cache grafting in latent space can also carry knowledge. For researchers focused on small-model deployment and RAG optimization, this is a signal worth following.

One-line Summary

CacheGraft replaces text RAG with offline fused KV caches, with a small model winning 29/30 in zero-shot comparisons.

Paragraph Summary

CacheGraft proposes a new RAG path that bypasses text chunks: based on the Cache-to-Cache (C2C) fusion framework, it encodes each corpus passage offline into a fused KV cache between a large model (4B) and a small model (0.6B), then injects that cache directly into the small model at query time without the large model being present. Across five question-answering benchmarks and two independent local judges, the method outperforms the pure-text receiver baseline in 29 of 30 comparisons, 17 of which are statistically significant; moreover, the gains are zero-shot for the fuser, indicating the C2C framework itself has generalization capability. The authors position the results as preliminary findings and argue that latent-space document enrichment is especially attractive for compute-constrained scenarios where only small models can be deployed at inference time, though cache compatibility, cross-model transfer, and scalability remain open problems.