VRAM Capacity
The storage capacity of dedicated memory on accelerators such as GPUs; it determines the number of model parameters a single device can hold and the size of the KV cache.
Timeline (last 90 days)
Going from 16G to 24G of VRAM, median decoding speed jumps from 56.8 tok/s to 98.3 tok/s, a 73% increase.
Memory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.
Under AI workloads, the bottleneck is usually VRAM bandwidth and capacity rather than temperature, unless overheating causes throttling.
All Facts (3)
Going from 16G to 24G of VRAM, median decoding speed jumps from 56.8 tok/s to 98.3 tok/s, a 73% increase.
50%UnverifiedMemory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.
50%UnverifiedUnder AI workloads, the bottleneck is usually VRAM bandwidth and capacity rather than temperature, unless overheating causes throttling.
50%