Gated DeltaNet
A recurrent model architecture that combines gating mechanisms with Delta-rule linear attention layers. It belongs to the family of linear attention hybrid designs and incurs low memory and compute overhead when processing long contexts.
Timeline (last 90 days)
Developers compiled a complete RISC-V CPU architecture into the model weights of Qwen 3.5 4B, using the Gated DeltaNet layers as an exact memory model, enabling programs to run without error through forward propagation alone, raising the arithmetic benchmark from 48% to a perfect 100%.
On RULER needle-in-a-haystack and long-context question answering tasks, MERLIN improves over the baseline by 8.3% on Gated DeltaNet-1.3B.
MERLIN was validated on multi-state variants of two architectures, Gated DeltaNet-1.3B and Mamba-2-780M.
On multiple LongBench subtasks, MERLIN improves by 13.5% on Gated DeltaNet-1.3B.
Gated DeltaNet updates its memory state based on the delta rule and is a modernized implementation of linear attention; Mamba-2 is a representative of state space models (SSM).
MERLIN works on both Gated DeltaNet and Mamba-2, validating its architecture-agnostic design claim.
KronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
Gated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
Mamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
NoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
All Facts (10)
Developers compiled a complete RISC-V CPU architecture into the model weights of Qwen 3.5 4B, using the Gated DeltaNet layers as an exact memory model, enabling programs to run without error through forward propagation alone, raising the arithmetic benchmark from 48% to a perfect 100%.
50%UnverifiedOn RULER needle-in-a-haystack and long-context question answering tasks, MERLIN improves over the baseline by 8.3% on Gated DeltaNet-1.3B.
50%UnverifiedMERLIN was validated on multi-state variants of two architectures, Gated DeltaNet-1.3B and Mamba-2-780M.
50%UnverifiedOn multiple LongBench subtasks, MERLIN improves by 13.5% on Gated DeltaNet-1.3B.
50%UnverifiedGated DeltaNet updates its memory state based on the delta rule and is a modernized implementation of linear attention; Mamba-2 is a representative of state space models (SSM).
50%UnverifiedMERLIN works on both Gated DeltaNet and Mamba-2, validating its architecture-agnostic design claim.
50%UnverifiedKronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
50%UnverifiedGated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
50%UnverifiedMamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
50%UnverifiedNoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
50%