Memory Bandwidth
A metric measuring the rate at which data is transferred between memory and the processor, typically expressed in GB/s. It represents the primary performance bottleneck during the decode phase when running large language models locally on a device.
Core Facts
Even after quantization, 70B-level models often require more than 40 GB of VRAM or a multi-GPU configuration.
Each time a token is generated, the Mac must read the entire model from memory, making memory bandwidth the fundamental upper limit on inference speed.
Running local AI models typically requires 8–16 GB or more of memory, a relatively recent CPU or a GPU-accelerated graphics card, and sufficient storage space.
Timeline (last 90 days)
The speed bottleneck of local LLM inference is often memory bandwidth rather than raw compute, because generating each token requires passing all weights through once.
Streaming inference must read and write cache state at every time step; if the state tensors are large or the access pattern is non-contiguous, bandwidth pressure quickly turns into latency.
Mobile CPUs typically share limited-bandwidth LPDDR memory, and memory bandwidth is an often overlooked bottleneck on Arm edge devices.
Three 200-megapixel sensors may cause three high-data-rate streams to work simultaneously, placing extremely high demands on ISP scheduling and memory bandwidth.
The theoretical upper limit formula for local AI token generation speed is: maximum speed (tokens/second) ≈ memory bandwidth ÷ model size.
With memory bandwidth unchanged, upgrading the CPU or increasing the number of cores will not substantially improve local large model inference speed.
The roughly 14% gap between the theoretical prediction of 36 tokens/second and the measured 31 tokens/second mainly comes from memory reads not reaching peak bandwidth, the computational overhead of the attention mechanism, and scheduling losses in the software stack.
You can roughly estimate the number of tokens generated per second by dividing model size by memory bandwidth.
For local inference, memory capacity determines "whether it can run," and memory bandwidth determines "how fast it runs."
Memory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
15 more timeline events
All Facts (20)
Even after quantization, 70B-level models often require more than 40 GB of VRAM or a multi-GPU configuration.
75%VerifiedEach time a token is generated, the Mac must read the entire model from memory, making memory bandwidth the fundamental upper limit on inference speed.
65%VerifiedRunning local AI models typically requires 8–16 GB or more of memory, a relatively recent CPU or a GPU-accelerated graphics card, and sufficient storage space.
65%Verified自回归大模型推理瓶颈并非计算量而是内存带宽，推测解码通过拒绝采样保证输出分布与原始大模型完全一致
65%VerifiedAutoregressive text generation is a memory-bandwidth-bound task, where memory bandwidth—not peak compute—is the true performance bottleneck.
65%Verified大语言模型的推理过程本质上是内存带宽受限（memory-bandwidth bound）的任务，每生成一个Token都需要将模型的全部权重从存储中读取一遍
65%UnverifiedThe speed bottleneck of local LLM inference is often memory bandwidth rather than raw compute, because generating each token requires passing all weights through once.
50%UnverifiedStreaming inference must read and write cache state at every time step; if the state tensors are large or the access pattern is non-contiguous, bandwidth pressure quickly turns into latency.
50%UnverifiedMobile CPUs typically share limited-bandwidth LPDDR memory, and memory bandwidth is an often overlooked bottleneck on Arm edge devices.
50%UnverifiedThree 200-megapixel sensors may cause three high-data-rate streams to work simultaneously, placing extremely high demands on ISP scheduling and memory bandwidth.
50%UnverifiedWith memory bandwidth unchanged, upgrading the CPU or increasing the number of cores will not substantially improve local large model inference speed.
50%UnverifiedThe theoretical upper limit formula for local AI token generation speed is: maximum speed (tokens/second) ≈ memory bandwidth ÷ model size.
50%UnverifiedThe roughly 14% gap between the theoretical prediction of 36 tokens/second and the measured 31 tokens/second mainly comes from memory reads not reaching peak bandwidth, the computational overhead of the attention mechanism, and scheduling losses in the software stack.
50%UnverifiedYou can roughly estimate the number of tokens generated per second by dividing model size by memory bandwidth.
50%UnverifiedFor local inference, memory capacity determines "whether it can run," and memory bandwidth determines "how fast it runs."
50%UnverifiedMemory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
50%UnverifiedInference speed (tokens/second) ≈ memory bandwidth ÷ model size
50%UnverifiedIn modern AI hardware design, moving data is often more costly than performing mathematical operations, and memory bandwidth frequently becomes the limiting factor
50%UnverifiedThe author of OpenTPU reports that the speed of token generation is limited by memory bandwidth rather than compute itself
50%UnverifiedOnline real-time inference is typically a low-latency task involving small batches or single requests, where the bottleneck is often memory bandwidth rather than compute.
50%