Causal Attention
The core self-attention mechanism of the Transformer decoder, which uses a lower-triangular mask to ensure that the token at position i can only attend to the prefix from 0 to i, forming the theoretical basis for token-by-token generation and prefix KV reuse.
Timeline (last 90 days)
在因果注意力机制下的自回归语言模型中，已缓存的 prefix 状态不会被后续 token 改写，缓存一旦写入便与完全重计算的结果完全一致。
In causal attention, positions at the front of the sequence participate in far less computation than positions at the back, which easily leaves threads idle.
The Transformer's causal attention mechanism dictates that each token's KV is related only to all the tokens before it, so prompts with completely identical prefixes generate completely identical KV that can be reused directly.
Prompt缓存必须从前缀开始严格匹配，一旦中间某个token变化，其后所有token的注意力计算都会改变，已缓存的KV对随之失效
All Facts (4)
在因果注意力机制下的自回归语言模型中，已缓存的 prefix 状态不会被后续 token 改写，缓存一旦写入便与完全重计算的结果完全一致。
50%UnverifiedIn causal attention, positions at the front of the sequence participate in far less computation than positions at the back, which easily leaves threads idle.
50%UnverifiedThe Transformer's causal attention mechanism dictates that each token's KV is related only to all the tokens before it, so prompts with completely identical prefixes generate completely identical KV that can be reused directly.
50%UnverifiedPrompt缓存必须从前缀开始严格匹配，一旦中间某个token变化，其后所有token的注意力计算都会改变，已缓存的KV对随之失效
50%