Adaptive KV Streaming
A KV cache management technique in llama.cpp. The inference program proactively schedules data transfers and uses a circular buffer for prefetching, enabling overlap between data transfer and computation. This allows long-context inference to run on devices with limited GPU memory.
Timeline (last 90 days)
llama.cpp Adaptive KV Streaming changes data movement from being managed automatically by the system to being actively scheduled by the inference program; the full KV is stored in memory, only a portion resides permanently in VRAM, and a circular buffer prefetches the KV needed by subsequent layers.
Adaptive KV Streaming retains the full history without relying on periodic context truncation to save space; the trade-off is that it consumes a large amount of system memory and its speed is limited by the memory-to-GPU transfer bandwidth.
This solution allows a GPU with 16GB of VRAM to run a 27B-class model with a 192K context and vision capability, but the speed ceiling is constrained by hardware bandwidth.
All Facts (3)
llama.cpp Adaptive KV Streaming changes data movement from being managed automatically by the system to being actively scheduled by the inference program; the full KV is stored in memory, only a portion resides permanently in VRAM, and a circular buffer prefetches the KV needed by subsequent layers.
50%UnverifiedAdaptive KV Streaming retains the full history without relying on periodic context truncation to save space; the trade-off is that it consumes a large amount of system memory and its speed is limited by the memory-to-GPU transfer bandwidth.
50%UnverifiedThis solution allows a GPU with 16GB of VRAM to run a 27B-class model with a 192K context and vision capability, but the speed ceiling is constrained by hardware bandwidth.
50%