Low-Bit Quantization Breakthrough: GGA+PW Makes Linear Attention Long-Text Generation More Accurate and Shorter

Linear attention models rely on fixed-size recurrent states to compress historical information in long-context reasoning. This design inherently saves storage, but once the state is quantized to low bits, quantization write-back errors accumulate across subsequent recurrence steps, degrading prediction quality on long reasoning tasks and even causing repetitive generation and overly long outputs. A recent study on low-bit GDN (Gated DeltaNet) recurrent-state quantization proposes two methods—Global Gradient-aware Allocation (GGA) and Pure-Write Sliding Window (PW)—aiming to simultaneously solve the challenges of precision allocation and recent-information maintenance within a storage budget.

Why Low-Bit State Quantization Gets "Worse the More You Write"

The recurrent state is the sole carrier of historical information in linear attention models. At each inference step, new information is written into the state while the state is read out to participate in prediction. When the state is compressed to low bits, write-back operations introduce quantization residuals. These residuals do not disappear; instead, they accumulate progressively with the number of recurrence steps. In long reasoning tasks, error accumulation directly weakens the model's ability to recall early context, manifesting as decreased answer accuracy, repetitive generated content, and the model outputting excessively long text without being able to "stop."

The research team points out that simply increasing the bit width of the accumulated state is not optimal: with the same storage budget, allocating it to maintain raw information from recent writes often yields higher task scores and shorter outputs. This observation forms the design starting point for both GGA and PW.

Linear Attention is an efficient variant of Transformer attention. Its core idea is to use kernel functions to decompose softmax attention into a linearly accumulable form, reducing inference complexity from quadratic in sequence length to linear. GDN (Gated DeltaNet) builds on this by introducing a gating mechanism and the Delta rule: gating controls the forgetting rate of historical information, while the Delta rule makes state updates approximate gradient steps in online learning, enabling the recurrent state to selectively remember or overwrite information. As a result, the GDN state is not merely a simple accumulator but carries weighted historical memory, with each write interacting with existing state. This also explains why quantization residuals are more harmful in GDN: residuals not only exist as noise but are amplified or persisted through gating and Delta updates, thereby distorting all subsequent readouts.

GGA: Using Gradient Sensitivity to Decide "Who Deserves More Bits"

The core idea of GGA is to combine quantization residuals with gradient sensitivity. Specifically, it evaluates the gradient impact of each attention head's candidate quantization residuals on future prediction loss, then allocates bit widths to different heads under a model-level mixed-precision budget. In other words, heads more sensitive to the final prediction receive higher precision, while insensitive heads are compressed more aggressively, reducing overall quantization error without changing the total budget.

According to the paper, on the Qwen3.5-4B and 9B models, using GGA allocation alone without any other modifications improved scores across seven benchmarks by 19.48 and 29.84 percentage points respectively compared to STEPQuant@4. This demonstrates that the precision allocation strategy itself can bring significant gains without additional storage overhead.

Mixed-Precision Quantization refers to allocating different bit widths to different parts within the same model rather than uniformly compressing globally. The key challenge lies in measuring the "importance" of each part. Gradient sensitivity is a commonly used criterion: apply a small perturbation to a parameter or activation and observe the first-order response of the loss function; a larger response indicates that the part is more critical to the final prediction. GGA transfers this idea to quantization residuals—evaluating "if this head's state is quantized, how much impact will its residuals have on the loss"—thereby tilting the limited bit budget toward highly sensitive heads. Compared to layer-wise or channel-wise heuristic allocation, gradient-aware allocation is closer to the task loss but requires computing gradients on calibration data, incurring some offline overhead.

PW: Giving Recent Writes a "Clean Channel"

PW addresses a different problem: recent information should not be mixed with low-bit historical state. The method independently maintains pure-write content from the most recent period, stored as FP8 vector pairs, and lets these vectors fully participate in the GDN state transition process. Only after they leave the sliding window are they merged into the low-bit historical state. This way, the model uses high-precision information when reasoning over recent context, while distant history remains stored long-term in low-bit form.

After adding PW-S8 on top of GGA, the seven-task macro averages reached 69.83% and 78.63%, improvements of 38.39 and 41.21 percentage points respectively over STEPQuant@4. As a detail, precision reallocation and window maintenance not only improve task scores but also alleviate the overly long output problem: under the @6 setting, GGA reduced average generation length (equally weighted across tasks) by 10.45% and 14.65% relative to STEPQuant; adding PW-S8 on top of GGA@4 further reduced average generation length by 32.32% and 36.77%.

Sliding Window is a common structure for maintaining recent context in streaming inference: only the raw information from the most recent steps is retained, while earlier content is compressed or discarded. What makes PW special is that it does not discard information outside the window; instead, it waits for these pure-write vectors to leave the window before merging them into the low-bit historical state, forming a hierarchical memory of "high precision for recent, low precision for distant." FP8 (8-bit floating point) is a low-precision format widely supported by hardware in recent years. Compared to INT8, it better preserves dynamic range and is suitable for representing write vectors with wide numerical distributions. The "8" in window size S8 refers to the number of steps or slots covered by the window. A larger window provides higher recent fidelity but requires more extra storage, so it must be balanced against the total budget.

How to Spend the Storage Budget More Wisely

The paper also includes comparisons under the same storage budget: allocating extra capacity to maintain recent writes yields higher task scores and shorter average outputs than simply increasing the bit width of the accumulated state. This conclusion has direct practical implications for deployment—in scenarios with limited GPU memory or bandwidth, prioritizing the precision of recent information is more effective than pursuing high bit widths for historical state.

Implications for Long-Context Reasoning

The value of this work lies in pushing low-bit quantization from "uniform compression" toward a combined strategy of "sensitivity-based allocation + temporal layering." GGA handles precision allocation in the spatial dimension, while PW handles information freshness in the temporal dimension; the two are complementary. For models that need to run long-context inference on edge devices or low-cost hardware, this approach provides a reusable engineering path: rather than raising precision across the board, spend resources where they matter most—keeping sensitive heads and recent memory at higher fidelity.

Of course, the paper has only validated seven benchmarks on Qwen3.5-4B/9B, and the method's generalization to different model architectures and longer contexts remains to be further observed. But based on the available data, the combination of GGA and PW brings considerable improvements in both task scores and generation length, offering a new reference for low-bit deployment of linear attention models.