PaperKIVI
KIVI
A KV cache quantization method for autoregressive LLMs that quantizes the Key cache per channel and the Value cache per token, exploiting the differing distributional characteristics of K and V.
A KV cache quantization method for autoregressive LLMs that quantizes the Key cache per channel and the Value cache per token, exploiting the differing distributional characteristics of K and V.