Causal Attention
The core self-attention mechanism of the Transformer decoder, which uses a lower-triangular mask to ensure that the token at position i can only attend to the prefix from 0 to i, forming the theoretical basis for token-by-token generation and prefix KV reuse.
Timeline (last 90 days)
In autoregressive language models under causal attention, cached prefix states are not rewritten by subsequent tokens, and once the cache is written it is exactly consistent with the result of full recomputation.
In causal attention, positions at the front of the sequence participate in far less computation than positions at the back, which easily leaves threads idle.
The Transformer's causal attention mechanism dictates that each token's KV is related only to all the tokens before it, so prompts with completely identical prefixes generate completely identical KV that can be reused directly.
Prompt缓存必须从前缀开始严格匹配，一旦中间某个token变化，其后所有token的注意力计算都会改变，已缓存的KV对随之失效
All Facts (4)
In autoregressive language models under causal attention, cached prefix states are not rewritten by subsequent tokens, and once the cache is written it is exactly consistent with the result of full recomputation.
50%UnverifiedIn causal attention, positions at the front of the sequence participate in far less computation than positions at the back, which easily leaves threads idle.
50%UnverifiedThe Transformer's causal attention mechanism dictates that each token's KV is related only to all the tokens before it, so prompts with completely identical prefixes generate completely identical KV that can be reused directly.
50%UnverifiedPrompt缓存必须从前缀开始严格匹配，一旦中间某个token变化，其后所有token的注意力计算都会改变，已缓存的KV对随之失效
50%