Product

FlexGen

A throughput-oriented inference system for large language models that supports offloading model weights and KV caches to CPU DRAM and SSD, enabling large-scale language models to run on low-cost hardware.

Timeline (last 90 days)

Sep 16

Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore attempt to extend GPU VRAM capacity using CPU DRAM and SSDs.

Unverified50%

All Facts (1)

Unverified

Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore attempt to extend GPU VRAM capacity using CPU DRAM and SSDs.

50%

Source Articles

Where Should KV Cache Live? A Deep Dive into GPU/CPU/SSD Tiered Placement StrategiesSep 16