TraceKV: Policy-Preserving Low-Rank KV Cache Compression Brings Near-Lossless Reasoning VLMs

TraceKV uses training-free low-rank KV compression to preserve reasoning VLM policies, achieving 5.64x compression with near-lossless accuracy.
TraceKV is a training-free low-rank KV cache compression framework for reasoning VLMs. It addresses policy drift from calibration state mismatch and unequal KV error sensitivity via on-policy calibration, sensitivity-aware SVD, and global rank allocation. Combined with quantization, it achieves 5.64x KV cache compression with near-lossless reasoning accuracy and up to 1.83x decoding speedup; code will be open-sourced.
Reasoning vision-language models (VLMs) face rapidly expanding KV cache demands when simultaneously handling large-scale multimodal contexts and ever-growing reasoning traces. Low-rank KV compression alleviates memory pressure by storing low-dimensional latent representations of KV states, and can be combined with quantization. However, existing low-rank methods often cause increasingly large predictive policy drift from the dense model when compressing reasoning KV states, ultimately leading to reasoning failures. TraceKV proposes a training-free low-rank KV compression framework that unifies policy calibration, sensitivity-aware SVD, and global rank allocation to substantially reduce policy KL divergence during inference, achieving near-lossless reasoning accuracy alongside significant cache compression.
Why Does Low-Rank Compression Fail on Reasoning VLMs?
The core idea of low-rank KV compression is to project high-dimensional Key/Value states into a low-dimensional latent space, thereby reducing storage and bandwidth. But the paper points out that when the compression target shifts from ordinary text KV to the KV states of reasoning VLMs, existing methods encounter two classes of problems.
The first is calibration state mismatch. The context distribution in static calibration datasets differs from the on-policy states the model actually produces during inference, causing the compressed model's behavior to deviate from the original dense model. The second is unequal error sensitivity across KV states. KV states at different positions and in different modalities have vastly different impacts on the final reasoning result; treating all states uniformly amplifies the loss of critical information.
Together, these two factors cause cumulative policy drift on reasoning tasks, manifesting as broken reasoning chains or incorrect answers.

KV cache is a key memory component in the Transformer decoding process. During autoregressive generation, each time a new token is generated, the model must cache the Key and Value vectors of all previous tokens so they can be reused in attention computation. For reasoning VLMs, this cache contains not only the KV of text tokens but also the KV of visual tokens, and the longer the reasoning chain, the larger the cache. Its memory footprint is roughly proportional to the product of the number of layers, number of attention heads, head dimension, sequence length, and batch size, so long contexts and multi-step reasoning jointly drive up GPU memory demands. Low-rank compression reduces storage by projecting high-dimensional KV into a low-dimensional latent space, but compression errors accumulate with decoding steps—which is precisely why reasoning scenarios are more sensitive than ordinary text generation.
TraceKV's Three Key Designs
TraceKV is a training-free framework that requires no additional model parameter fine-tuning. Instead, it starts from calibration and decomposition strategies, unified under the principle of "policy preservation" to address the above problems.
On-policy Calibration
To address calibration state mismatch, TraceKV adopts on-policy calibration—using the states the model actually produces during inference to calibrate the low-rank decomposition, rather than relying on static datasets. This makes the compressed representation closer to the model's true reasoning behavior and reduces distribution shift.
On-policy originates from reinforcement learning, referring to learning or calibrating using trajectories actually produced by the current policy rather than relying on offline fixed datasets. In the context of KV compression, on-policy calibration means letting the model reason freely for a while first, collecting the KV states generated during its real decoding process, and then solving the low-rank decomposition based on these states. In contrast, off-policy calibration uses KV obtained from forward passes over static corpora; the two distributions differ: when a reasoning model generates intermediate steps, the input distribution changes with its own outputs, and static data cannot cover these states. Therefore, on-policy calibration can narrow the distribution shift between the model before and after compression—this is also the foundation for TraceKV achieving policy preservation.
Sensitivity-Aware SVD
To address unequal error sensitivity, TraceKV introduces a sensitivity-aware mechanism into the SVD decomposition process, giving higher reconstruction priority to the more critical parts of KV states. The paper combines this idea with global rank allocation, dynamically allocating ranks to different layers or states under an overall rank budget constraint, thereby retaining more policy-important information at the same compression rate.
SVD (Singular Value Decomposition) is a common tool for low-rank compression: after decomposing the KV matrix, retaining the largest singular values and corresponding vectors yields the optimal reconstruction for a given rank. But standard SVD minimizes reconstruction mean squared error and does not distinguish the impact of different KV states on the final output. Sensitivity-aware SVD introduces weighting or importance measures during decomposition, giving higher priority to directions that have greater impact on the policy. Rank allocation determines how much rank to assign to each layer, attention head, or modality when the total rank budget is fixed. Global rank allocation models this as an overall optimization rather than independently setting a fixed rank per layer, thereby concentrating rank at more sensitive positions at the same compression rate and reducing policy KL divergence.
Global Rank Allocation
Global rank allocation means no longer using a fixed rank for each layer, but optimizing rank allocation globally based on sensitivity. Together with on-policy calibration and sensitivity-aware SVD, this strategy forms TraceKV's "policy preservation" framework, with the goal of minimizing the policy KL divergence between the compressed model and the dense model.
Experimental Results: Near-Lossless Reasoning and Significant Speedup
The paper conducts experiments on multiple VLMs and visual reasoning benchmarks. Results show that TraceKV outperforms existing low-rank methods in the accuracy-memory trade-off. When combined with quantization, TraceKV achieves 5.64x KV cache compression while maintaining near-lossless reasoning accuracy. In addition, the optimized low-rank attention kernel delivers up to 1.83x end-to-end decoding speedup.
These data indicate that by unifying calibration, decomposition, and rank allocation under the policy preservation principle, it is possible to substantially compress the KV cache while avoiding a significant decline in reasoning capability. For multimodal applications requiring long reasoning chains, this method is expected to reduce GPU memory usage and improve decoding efficiency.
Implications for Reasoning VLM Deployment
Reasoning VLMs are becoming important tools for complex visual question answering and multi-step reasoning tasks, but their KV cache overhead rises rapidly with context length and reasoning steps. TraceKV's value lies in the fact that it does not require retraining the model; instead, it reduces memory usage directly at inference time through smarter compression strategies. A 5.64x compression rate means longer contexts or larger batches can be supported on the same hardware, while the 1.83x decoding speedup directly improves user experience.
The paper's code will be open-sourced on GitHub at https://github.com/AK-DREAM/TraceKV; its actual deployment performance and generalization across different models are worth following.
For researchers and engineers focused on multimodal reasoning efficiency, TraceKV offers a direction worth attention: compression should consider not only reconstruction error but also the impact on the model's predictive policy. The policy-preservation perspective may become a new consensus for reasoning model compression.
One-line Summary
TraceKV uses training-free low-rank KV compression to preserve reasoning VLM policies, achieving 5.64x compression with near-lossless accuracy.
Paragraph Summary
TraceKV is a training-free low-rank KV cache compression framework for reasoning vision-language models. The article notes that existing low-rank methods in reasoning scenarios suffer from calibration state mismatch and unequal error sensitivity across KV states, causing cumulative policy drift between the compressed and dense models that ultimately leads to reasoning failures. TraceKV takes "policy preservation" as its unifying principle: on-policy calibration aligns the decomposition with the model's true reasoning states, sensitivity-aware SVD gives higher reconstruction priority to critical KV, and global rank allocation optimizes per-layer ranks under a total budget. Experiments show that combined with quantization it achieves 5.64x KV cache compression while maintaining near-lossless reasoning accuracy, and the optimized kernel delivers up to 1.83x end-to-end decoding speedup. Code will be open-sourced.
Related articles

US Classified National Security Communications Network Outage: A Security Warning for the AI Era
Reuters reports a highly classified US national security communications network outage, sparking Hacker News discussion. This article analyzes critical infrastructure resilience challenges and their implications for AI system security design and failure recovery.

SEC Investigates Nvidia: Behind the Jensen Huang Departure Controversy
The Wall Street Journal reports the SEC is investigating Nvidia following CEO Jensen Huang's departure and admitted affair with an alleged Chinese spy. This article maps the timeline, compliance risks, and potential impact on the AI chip industry, while noting the information awaits cross-verification.

MacBook Neo Sees Rare $50 Discount Ahead of Black Friday — A Good Time for Students to Buy
MacBook Neo gets a rare $50 discount ahead of Black Friday: 256GB at $649, 512GB at $749, all four colors available at Amazon, Best Buy, and Target—great for students.