Mamba-2

SSM is a class of architectures that process sequences in a linear recurrent manner, with representatives such as Mamba and Mamba-2.

MERLIN works on both Gated DeltaNet and Mamba-2, validating its architecture-agnostic design claim.

Gated DeltaNet updates its memory state based on the delta rule and is a modernized implementation of linear attention; Mamba-2 is a representative of state space models (SSM).

On multiple LongBench subtasks, MERLIN improves by 12.0% on Mamba-2-780M.

MERLIN was validated on multi-state variants of two architectures, Gated DeltaNet-1.3B and Mamba-2-780M.

The industry already has hybrid backbone practices such as Mamba-2/Transformer hybrids and Jamba.

The forward computation of Mamba-2 includes selective scan, state recurrence and several element-wise gating operations, with both serial dependencies along the sequence dimension and parallel space along the feature dimension.

The NVIDIA blog describes the core idea of the Mamba-2 forward kernel as splitting the sequence into blocks, computing local contributions in parallel within each block, and connecting blocks via state passing.

cuTile's tile semantics can express the structure of "parallel within a block, recurrence between blocks," letting the scan process advance in pipelines of tiles while keeping state resident in on-chip storage, reducing round trips to global memory.

Nemotron 3.5 Super VL uses a Mamba-2/Transformer hybrid MoE architecture, with 512 routed experts per MoE layer, paired on the vision side with a C-RADIO encoder, supporting text, JPEG/PNG images, and MP4 video up to 2 minutes long, with a context limit of 1M tokens and native support for multi-token prediction.

Mamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.

Mamba-2 builds on selective state space models by introducing a more structured state transition matrix, expressing the state update in matrix multiplication form, thereby achieving higher throughput during training.

NoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.

Nemotron 3.5 Bolt's architecture adopts LatentMoE mixing Mamba-2 and Transformer, and the vision side uses the C-RADIO v4-H encoder.

Nemotron 3.5 Lightning uses a hybrid structure that interleaves Mamba-2 with MoE layers, paired with a small number of attention layers.

The SSD (State Space Duality) framework proposed by Mamba-2 reveals that SSMs with a specific structure are mathematically equivalent to a masked form of linear attention, allowing training to be parallelized using matrix multiplication.

Zamba2 (Zyphra) is designed as a Mamba-2 backbone plus shared-weight attention blocks inserted every N layers.

NVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.

Most SSM and linear attention models use a linear attention mechanism at the bottom layer, with Mamba-2 implemented through the SSD structure and xLSTM implemented through covariance updates.

Falcon-H1 runs attention heads and Mamba-2 heads in parallel within the same block, rather than replacing them layer by layer.