State Space Model

SSM is a class of architectures that process sequences in a linear recurrent manner, with representatives such as Mamba and Mamba-2.

State space models represented by Mamba compress historical information by maintaining a fixed-size hidden state, with both time and space complexity of O(1) during inference.

Mamba 等基于状态空间模型（SSM）的架构以线性复杂度实现序列建模，更适合实时推理

State space models (SSMs) such as Mamba abandon the attention mechanism and use recurrent structures to achieve linear time complexity sequence modeling.

Sub-quadratic loop alternatives (such as RNNs and SSMs) have compute that grows linearly with sequence length (O(n·d)), but limited state capacity may degrade long-range retrieval quality.

Early SSMs performed limitedly on language tasks, mainly because their state update method was input-independent, making it difficult to selectively attend to information as flexibly as attention.

Linear attention or SSM layers can process long sequences with near-linear complexity, but they are often weaker than full attention on precise recall and complex reasoning.

The state-space-model-based multi-scale classifier proposed in the paper achieved a macro F1 of 0.894 and a Gleason 5 F1 of 0.802 on the PANDA dataset using 5-fold cross-validation.

The state recurrence of an SSM is essentially a composition of affine transformations, and the composition of affine transformations is associative, so scan algorithms can be applied.

The operator patterns of architectures such as MoE and state space models differ markedly from those of the traditional Transformer, placing higher demands on compiler operator coverage and dynamic shape support, and this is where shortcomings are most easily exposed when adapting dedicated accelerators.

The Mamba2 architecture has linear time complexity and a constant-size state representation, and its state representation does not grow with sequence length.

State space models (SSMs) such as the Mamba series follow the same recurrent paradigm: a fixed state cannot losslessly retain all historical details, and earlier information gets overwritten by subsequent updates.

To maintain usable efficiency at the tens-of-millions context scale, it is usually necessary to introduce optimizations such as sparse attention, chunked retrieval, state space models or hybrid architectures.

Typical representatives of state space models include S4, Mamba and others, which write the recurrence as a continuous differential equation or discrete linear recurrence and achieve efficient parallel training with the help of convolution or selective scan.

Priming面临的第一个质疑是：跨架构的权重映射不是总能找到合理对应关系，如果目标架构引入了全新计算模块（如状态空间模型层），预训练权重就没有现成组件可以填进去。

The study finds that the output geometries of Transformers, state space models, and recurrent neural networks are aligned to a far greater degree than their activation geometries.

State space models (SSMs) such as Mamba completely abandon the attention matrix and use recurrent hidden states to achieve O(N) complexity, but still fall short on tasks requiring precise key-value retrieval.

The paper conducts theoretical matching experiments covering three architectural classes—contractive state-space models, gated recurrent models, and attention models—and the experimental results reproduce the state and stability laws predicted by theory.

State Space Models (SSMs) pull the direction back toward fixed-size recurrent memory, but use a different state structure and update rules.

The mathematical roots of SSMs come from linear time-invariant systems in control theory, using matrices A, B, and C to describe how inputs update the hidden state and how the hidden state maps to outputs.