MLA
A multi-head latent attention architecture designed by DeepSeek. The DSpark speculative decoding algorithm performs targeted engineering optimizations specifically tailored to the characteristics of this architecture.
Core Facts
DeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
MLA can reduce the KV Cache VRAM usage to 5%–13% of traditional multi-head attention.
Multi-Head Latent Attention (MLA) is an attention mechanism proposed by DeepSeek-V2.
Timeline (last 90 days)
The industry commonly uses architectural approaches such as GQA and MLA to compress the KV cache.
KV sharing can be stacked with memory-efficient sequence mixers (such as multi-head latent attention MLA and Kimi delta attention), and is orthogonal to them.
Flash MLA is a GPU operator specifically optimized for reading and attention computation on the MLA compressed KV cache format, using paging management similar to virtual memory.
Models using GQA (grouped-query attention) or MLA (multi-head latent attention) can significantly compress the KV Cache, which is the key architectural prerequisite for running long contexts on small VRAM.
Deploying V4.1 Flash 1017 requires enabling paged attention (e.g., vLLM); otherwise, internal KV cache fragmentation erases all the gains brought by the MLA architecture.
Because the Max configuration uses MLA to compress the KV cache, it can fit a larger expert array into HBM3e memory, increasing the equivalent batch size by 30% to 40% compared with similar models.
WordCraft's citation features support tables of contents, footnotes, citations in APA, MLA, Chicago, and IEEE formats, as well as indexes and captions.
TokenSpeed's MLA decode kernel has about half the latency of TensorRT-LLM under typical speculative decoding workloads.
TokenSpeed's MLA kernel optimized for the Blackwell architecture has been adopted by vLLM.
MLA projects K and V into a latent vector space through low-rank joint compression, and at inference time only the compressed latent vectors are cached, compressing the KV cache several times over while maintaining expressive capability.
13 more timeline events
All Facts (20)
DeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
80%VerifiedMLA can reduce the KV Cache VRAM usage to 5%–13% of traditional multi-head attention.
75%VerifiedMulti-Head Latent Attention (MLA) is an attention mechanism proposed by DeepSeek-V2.
75%VerifiedGQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%VerifiedMLA compresses KV pairs into a low-dimensional latent space for caching and recovers the full Keys and Values via an up-projection matrix during inference, compressing the KV cache by several times.
75%VerifiedDeepSeek released DeepSeek-V4.1-Flash, a method that significantly compresses the memory required for KV cache.
70%UnverifiedMLA中RoPE只对部分维度施加旋转编码，采用分离式设计将一小部分维度专门用于承载RoPE位置编码，与压缩后的内容向量拼接使用
85%UnverifiedDeepSeek V4系列采用了MLA、MoE路由优化以及NSA等创新架构技术
85%UnverifiedThe quantized MLX KV cache reduces prompt memory usage by up to 74% compared to before.
70%UnverifiedDeepSeek-V2 proposes MLA (Multi-head Latent Attention), which compresses KV Cache to approximately 5% of MHA through low-rank compression.
70%UnverifiedDeepSeek's prefix caching implementation is based on the MLA (Multi-head Latent Attention) architecture, which reduces the storage requirements of the KV Cache by tens of times through low-rank compression.
60%UnverifiedThe industry commonly uses architectural approaches such as GQA and MLA to compress the KV cache.
50%UnverifiedKV sharing can be stacked with memory-efficient sequence mixers (such as multi-head latent attention MLA and Kimi delta attention), and is orthogonal to them.
50%UnverifiedFlash MLA is a GPU operator specifically optimized for reading and attention computation on the MLA compressed KV cache format, using paging management similar to virtual memory.
50%UnverifiedModels using GQA (grouped-query attention) or MLA (multi-head latent attention) can significantly compress the KV Cache, which is the key architectural prerequisite for running long contexts on small VRAM.
50%UnverifiedDeploying V4.1 Flash 1017 requires enabling paged attention (e.g., vLLM); otherwise, internal KV cache fragmentation erases all the gains brought by the MLA architecture.
50%UnverifiedBecause the Max configuration uses MLA to compress the KV cache, it can fit a larger expert array into HBM3e memory, increasing the equivalent batch size by 30% to 40% compared with similar models.
50%UnverifiedWordCraft's citation features support tables of contents, footnotes, citations in APA, MLA, Chicago, and IEEE formats, as well as indexes and captions.
50%UnverifiedTokenSpeed's MLA decode kernel has about half the latency of TensorRT-LLM under typical speculative decoding workloads.
50%UnverifiedModels using MLA such as DeepSeek-V4.1-Flash compress the KV state to about 2.4 GB per 32k stream (FP8 KV), while wider architectures consume 3.0 GB–3.8 GB per 32k stream.
50%