Selective Scan
Selective scan is the core operation of the Mamba family of state space models. Its discretization parameters vary dynamically with each token, enabling the model to retain or forget historical states based on content, with forward computation recurring along the sequence dimension.
Timeline (last 90 days)
Selective scan is the core operation of the Mamba family of SSMs, and its discretization parameter Δ, input matrix B and output matrix C all change dynamically with each token.
The forward computation of selective scan is a recurrence along the sequence dimension: h_t = A_bar_t · h_{t-1} + B_bar_t · x_t, where A_bar and B_bar are determined by Δ.
Because the recurrence has a strict temporal dependency, selective scan cannot compute all positions in parallel at once as attention does, and must use blocked scan or parallel prefix sum algorithms to break the serial chain into two stages: parallelism within blocks and transfer between blocks.
All Facts (3)
Selective scan is the core operation of the Mamba family of SSMs, and its discretization parameter Δ, input matrix B and output matrix C all change dynamically with each token.
50%UnverifiedThe forward computation of selective scan is a recurrence along the sequence dimension: h_t = A_bar_t · h_{t-1} + B_bar_t · x_t, where A_bar and B_bar are determined by Δ.
50%UnverifiedBecause the recurrence has a strict temporal dependency, selective scan cannot compute all positions in parallel at once as attention does, and must use blocked scan or parallel prefix sum algorithms to break the serial chain into two stages: parallelism within blocks and transfer between blocks.
50%