Continuum Attention
An attention paradigm that generalizes attention from a discrete set of key-value points to a continuous memory curve. It integrates over a relevance density function and computes the normalized attention output in closed form, preserving causality and quadratic complexity.
Timeline (last 90 days)
Continuum Attention (CA) is a new attention mechanism that treats key-value storage as a continuous piecewise linear memory curve rather than a discrete set of points, computing the normalized attention output in closed form via integration and interpolation.
Continuum Attention retains the same quadratic sequence complexity as standard attention; it is not an efficiency optimization scheme but an exploration of a theoretical paradigm.
In Continuum Attention, because the relevance density takes an exponential form, the integral operation reduces to a closed-form expression of exponential moments, and gradients can be derived analytically from at most second-order moments.
On the validation-set average NLL metric, Continuum Attention reduces by 0.0200 nats/byte compared to ordinary softmax attention.
On a non-overlapping validation slice, Continuum Attention achieves a 0.0457 nats/byte advantage over the strongest softmax baseline (a screened version), accompanied by a paired 95% confidence interval.
The paper's authors point out that these findings motivate continuous integration as a promising alternative to attention, but at the same time acknowledge that large-scale generalization ability remains to be verified.
All Facts (6)
Continuum Attention (CA) is a new attention mechanism that treats key-value storage as a continuous piecewise linear memory curve rather than a discrete set of points, computing the normalized attention output in closed form via integration and interpolation.
50%UnverifiedContinuum Attention retains the same quadratic sequence complexity as standard attention; it is not an efficiency optimization scheme but an exploration of a theoretical paradigm.
50%UnverifiedIn Continuum Attention, because the relevance density takes an exponential form, the integral operation reduces to a closed-form expression of exponential moments, and gradients can be derived analytically from at most second-order moments.
50%UnverifiedOn the validation-set average NLL metric, Continuum Attention reduces by 0.0200 nats/byte compared to ordinary softmax attention.
50%UnverifiedOn a non-overlapping validation slice, Continuum Attention achieves a 0.0457 nats/byte advantage over the strongest softmax baseline (a screened version), accompanied by a paired 95% confidence interval.
50%UnverifiedThe paper's authors point out that these findings motivate continuous integration as a promising alternative to attention, but at the same time acknowledge that large-scale generalization ability remains to be verified.
50%