KV Cache Quantization Benchmarked: 8-bit Is Nearly Lossless, How Much Does 4-bit Cost?

KV cache quantization has long been a contentious topic in local LLM deployment. Some insist on "never quantize," while others believe "4-bit is good enough." A recent systematic test on Qwen3.8 27B from the Reddit community used KL divergence (KLD) to quantify the impact of different KV cache precisions on model output distributions, covering three major inference engines—llama.cpp, exllamav3, and vLLM—and testing 35 weight quantization variants, each paired with 16-bit, 8-bit, and (where supported) 4-bit KV cache. The conclusion is clear: 8-bit KV cache is essentially free, while 4-bit requires careful trade-offs.

Test Methodology: Measuring Output Drift with KL Divergence

The author used token-by-token KL divergence to measure the difference in probability distributions between quantized models and the original bf16 model, where 0 means perfect agreement. Top-1 agreement rate was also tracked—how often both models select the same most likely token. To make results comparable across engines, the author first ran unquantized models on both llama.cpp and vLLM as a control; the two differed by only 0.0004 in KLD, equivalent to the difference between two correct implementations in floating-point operation order. This confirms that cross-engine KLD comparisons are valid.

The dataset was also reworked this time, mixing agentic tool-calling samples generated by full-precision Qwen3.8 27B with text from multiple domains, with single-line contexts extending up to 64k tokens. Each quantized variant was scored against an unquantized bf16 reference model: GGUF and EXL3 against Bartowski's bf16 GGUF, and vLLM checkpoints against the official bf16 safetensors.

KL Divergence (Kullback-Leibler Divergence) measures the difference between two probability distributions: if distribution Q is used to approximate the true distribution P, KLD quantifies the average extra information cost (in bits or nats) per event. It heavily penalizes cases where "P has probability but Q is nearly zero," making it particularly suitable for detecting whether a quantized model has dropped tokens that should have appeared. In practical LLM output comparisons, KLD is typically computed over the full vocabulary distribution at each position and then averaged; smaller values indicate the output distribution is closer to the original model. Note that KLD is asymmetric—P→Q does not equal Q→P. This article consistently uses the bf16 original model as reference distribution P and the quantized model as Q. Since KLD is a relative measure, absolute values cannot be directly compared across different models or datasets, which is why the author first verified cross-engine comparability.

bf16 (bfloat16) is a 16-bit floating-point format with the same bit width as the common fp16, but with the exponent increased from 5 to 8 bits and the mantissa reduced to 7 bits. Having the same exponent width as fp32 means bf16 has the same dynamic range as single-precision floating point (approximately 1e-38 to 3e38), avoiding the overflow at extreme values that fp16 suffers from, at the cost of lower effective precision (about 2-3 decimal digits). For neural network inference, dynamic range is typically more critical than mantissa precision, which is why bf16 has become the default reference precision for training and inference, and the baseline for all KLD comparisons in this article.

Weight Quantization: vLLM Checkpoints Still Lag Behind

Continuing from the Qwen3.6 test results, every vLLM checkpoint's KLD performance was beaten by smaller GGUF or EXL3 quantization variants. Part of the reason lies in activation quantization: NVIDIA and Unsloth's NVFP4 checkpoints run MLP layers as W4A4 (4-bit activations) with FP8 W8A8 for attention layers; Qwen's FP8 uses W8A8 throughout.

The author re-ran NVIDIA NVFP4 and Qwen FP8 using vLLM's Marlin kernel, keeping quantized weights but retaining bf16 activations: NVFP4's KLD dropped from 0.0256 to 0.0117, more than halving; FP8 went from 0.0039 to 0.0028, about a 30% reduction. Even so, NVFP4 only matches UD-Q5_K_XL, which is 8.7 GiB smaller. This result suggests that activation quantization may have a greater impact on output distributions than the weights themselves.

W4A4, W8A8, and NVFP4 describe the quantization bit widths for weights and activations respectively: W stands for Weight, A for Activation, and the number is the bit count. W8A8 means both weights and activations are represented in 8-bit integer/float, a relatively mature deployment scheme; W4A4 compresses both to 4 bits, offering higher compression but greater sensitivity to numerical error. NVFP4 is NVIDIA's 4-bit floating-point micro-scaling format using two-level scaling (one FP8 scale factor per 16-element block, plus a global scale), offering better precision within small blocks compared to formats like MXFP4, but it quantizes both weights and activations by default. The Marlin kernel is a high-performance GEMM implementation in vLLM for low-bit weights, supporting a "weight-only quantization" path that keeps weights at 4 bits while retaining bf16 activations—this is exactly the technique the author relied on to halve NVFP4's KLD in the re-run experiment. Clearly, for the same quantized weights, whether activations are quantized significantly changes the output distribution.

KV Cache Quantization: 8-bit Is Essentially Free

First, the memory savings. Taking 64k context as an example, 16-bit cache uses 64 KiB per token, totaling 4.00 GiB; 8-bit cache (llama.cpp's q8_0, EXL3 8-bit, vLLM's fp8_e4m3) uses about 32-34 KiB per token, totaling about 2 GiB, saving roughly 2 GiB; 4-bit cache (llama.cpp's q4_0, EXL3 4-bit) uses about 18 KiB per token, totaling 1.13 GiB, saving 2.9 GiB.

The 8-bit cache in llama.cpp and ExLlamaV3 comes at almost no cost. Across all tested quantized variants, the KLD change from q8_0 and EXL3 8-bit cache was within ±0.00012, with some quantizations even measuring slightly better, indicating this is entirely within numerical noise. vLLM's FP8 cache shows a slight increase, adding 0.0004 to 0.0006 KLD per checkpoint—still extremely small, but consistently appearing on almost every prompt. The author speculates this may be because enabling FP8 cache causes vLLM's attention backend to switch from FlashAttention to FlashInfer.

4-bit KV Cache: Measurable Cost, Strongly Correlated with Weight Precision

The KLD penalty from llama.cpp's q4_0 cache is roughly fixed, adding +0.0016 to +0.0027 across the tested GGUF quantizations. But this absolute increase hurts more precise weights disproportionately: on Q3-level quantizations (KLD around 0.025-0.035) it adds only 6-10%; on UD-Q4_K_XL it adds about 39%; on Q8_0 it adds about 390%—KLD jumps from 0.0007 to 0.0034, actually worse than UD-Q5_K_XL with 16-bit cache.

EXL3's 4-bit cache penalty clearly depends on weight precision: 2.0-2.5 bpw weights add only +0.0002 to +0.0004; 3.0 bpw adds about +0.0021 to +0.0022; 4.0 bpw adds +0.0026 to +0.0028; 6.0 and 8.0 bpw reach +0.0032 to +0.0033. In other words, if you're already using low-bit weights, the marginal cost of 4-bit KV cache is relatively acceptable; but if you're pursuing high-precision weights, 4-bit cache directly erases the benefits of weight quantization.

The Longer the Context, the Greater the 4-bit Penalty

The author scored 12 long documents at each position and found that the 4-bit cache penalty grows with context depth. Taking Q8_0 weights + q4_0 KV as an example, the KLD increase in the first 4k tokens is +0.0035, rising to +0.0041 in the 16-32k range, and reaching +0.0044 for 32-64k documents. Overall, the extra KL for 16-32k contexts is about 15-30% higher than for the first 4k. With the same quantized variants paired with 16-bit cache, KLD is unaffected across context lengths, clearly indicating the increase comes from the 4-bit cache itself. The author expects contexts beyond 128k to be even worse.

KV cache is the mechanism in autoregressive inference that caches the Key and Value tensors of historical tokens, avoiding recomputation of the entire context for each new token generated. Its memory usage grows linearly with context length: the cache size per token is approximately 2 × number of layers × number of KV heads × head dimension × bytes per element, so at long contexts the KV cache often rivals or exceeds model weights in size. Quantizing the KV cache means compressing these cached elements from 16 bits to 8 or 4 bits, directly reducing memory proportionally. But the KV cache differs from weights: it participates in attention's dot-product computation, so quantization errors are amplified by softmax and propagated layer by layer, and errors in historical tokens persistently affect all subsequent generation—this is the mechanistic reason why the 4-bit penalty accumulates with depth at long contexts. llama.cpp's q8_0 and q4_0 refer to its general block quantization formats, where every 32 elements share a scale factor; vLLM's fp8_e4m3 is 8-bit floating point (4-bit exponent, 3-bit mantissa), whose dynamic range is better suited to activation-like data.

Practical Recommendations: Choose KV Strategy Based on Weight Precision

Based on the combined data, the logic for choosing KV cache precision should be tied to weight precision:

• Using Q5 or higher precision weights: Prefer 8-bit KV cache. It's essentially lossless on llama.cpp and ExLlamaV3, with minimal cost on vLLM, directly saving half the cache memory. 4-bit cache isn't worth it, as it significantly undermines the advantages of high-precision weights.
• Using Q3 or lower precision weights: 8-bit remains recommended; 4-bit can serve as a fallback when memory is extremely tight, but you must accept gradually accumulating distribution drift at long contexts.
• Long-context scenarios: Be wary of 4-bit KV cache. The author notes that the KLD from KV cache quantization may differ from weight quantization KLD in terms of quality trade-offs, and community experience suggests that at long contexts, KV quantization may harm model performance more than weight quantization.

The final summary chart compares total memory (weights + 64k KV cache) against KL divergence, with the recommendation line clearly excluding 4-bit KV. For most local deployment scenarios, 8-bit KV cache is currently the best value choice—half the memory, almost no cost.

One-line Summary

Systematic Qwen3.8 27B testing shows: 8-bit KV cache is nearly lossless, while 4-bit requires careful trade-offs based on weight precision.

Paragraph Summary

Based on a systematic Reddit community test of Qwen3.8 27B, this article uses KL divergence and Top-1 agreement to measure the impact of different KV cache precisions on output distributions, covering llama.cpp, exllamav3, and vLLM across 35 weight quantization variants. Results show: 8-bit KV cache changes KLD by within ±0.00012 on llama.cpp and ExLlamaV3, essentially free with half the memory; vLLM's FP8 cache shows only a slight 0.0004-0.0006 increase. 4-bit cache brings measurable penalties, hurting high-precision weights more—KLD increases about 390% on Q8_0 versus only 6-10% at Q3 level, and grows with context, being 15-30% higher at 16-32k than the first 4k. For weight quantization, vLLM checkpoints still lose to smaller GGUF/EXL3, largely due to activation quantization; switching to the Marlin kernel while keeping bf16 activations halves NVFP4's KLD. The author recommends preferring 8-bit KV cache for Q5 and above weights.