MSLA
A linear attention variant that partitions long sequences into multiple segments, compresses each segment into an independent fixed-size memory state, and interacts only with the compressed states at query time, avoiding the capacity bottleneck of a single state while retaining linear complexity.
Timeline (last 90 days)
When aggregating information across memory states, MSLA relies only on the average alignment between the query and each segment's key, ignoring the distribution of alignment within the segment.
MERLIN tracks both the first and second moments of each segment's key, enabling the query to recover the mean and variance of alignment with each segment.
All Facts (2)
When aggregating information across memory states, MSLA relies only on the average alignment between the query and each segment's key, ignoring the distribution of alignment within the segment.
50%UnverifiedMERLIN tracks both the first and second moments of each segment's key, enabling the query to recover the mean and variance of alignment with each segment.
50%