Concept统一内存推理 / 显存内存混合推理
CPU Offloading
A large model deployment strategy that distributes model weights across GPU VRAM and system RAM on a per-layer basis, transferring layers on demand during inference. This approach enables local execution of models that exceed the memory capacity of a single GPU.
Source Articles
How Does VRAM Determine Whether a Model Can Run? A Deep Dive into VRAM CapacityOct 195 Seconds Without Adblock: A DDR4 Dating ComedyOct 14Running LLMs Locally on 8GB VRAM: A Model Selection Guide for Budget HardwareOct 3Running an 84GB LLM on a Single 7900XTX: 100+ Token/s Benchmark and Tuning BreakdownOct 3