ResQAdam: A Residual-Preserving Adaptive Optimizer That Substantially Cuts QAT Memory Overhead

Title

ResQAdam: A Residual-Preserving Adaptive Optimizer That Substantially Cuts QAT Memory Overhead

Meta Description

By preserving the sign information of QAT momentum residuals, the ResQAdam optimizer achieves accuracy close to AdamW within a low-rank framework while cutting optimizer state memory by 71%–74%, with less than 2% throughput loss.

Article Content

Quantization-aware training (QAT) is the most reliable way to obtain high-accuracy low-bit large language models (LLMs), but training with AdamW requires maintaining two dense FP32 states for every weight matrix, incurring enormous memory overhead. Recently, a paper titled "ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training" proposed a new optimizer, ResQAdam, aimed at solving this pain point.

The Memory Bottleneck of QAT and the Limitations of Low-Rank Optimizers

QAT lets models adapt to low-precision representations by simulating quantization error during training. However, the AdamW optimizer needs to store first- and second-order momentum for every parameter, both in FP32 precision, which causes training memory requirements to surge. Low-rank memory-efficient optimizers (such as GaLore and LDAdam) reduce state storage by projecting gradients into a low-rank subspace, but their performance drops significantly in QAT scenarios.

The authors found that a considerable portion of the QAT momentum lies outside its top-r singular subspace, and projection-based methods discard this information at every step. More crucially, they further observed that this information remains useful even if only the sign of the residual is retained. This finding provided the inspiration for designing a new optimizer.

Singular value decomposition and the principal singular subspace: After performing singular value decomposition (SVD) on the gradient (or momentum) matrix, one obtains a set of directions ordered by singular value from largest to smallest. The space spanned by the directions corresponding to the top r largest singular values is called the principal (top-r) singular subspace. The core assumption of low-rank optimizers is that the most important information during optimization is concentrated in this subspace, so storing and updating states only within it can greatly save memory. However, in QAT scenarios, gradient perturbations caused by quantization error distribute part of the useful information outside the principal singular subspace—the so-called "residual" part. Projection-based methods truncate this information at every step, and the accumulated effect leads to accuracy loss. The key observation of ResQAdam is precisely that although the residual cannot be fully stored, its sign direction still carries valid information; retaining the sign can recover most of the loss with almost no additional memory.

The precision difference between BF16 and FP32: BF16 (bfloat16) and FP32 are 16-bit and 32-bit floating-point formats, respectively, but BF16 has only an 8-bit mantissa (FP32 has 23 bits); its dynamic range matches FP32 while its precision is much lower. Changing optimizer states from FP32 to BF16 can directly halve this portion of memory, a common technique in memory-efficient training. However, BF16 momentum tends to lose low-order information when accumulating tiny updates, so compensation strategies such as normalization and sign-based updates are usually required. This is also one reason why ResQAdam applies Adam updates only to the principal-subspace coordinates and retains only the sign for the residual.

ResQAdam's Design: Adaptive Updates That Preserve the Residual Sign

Based on these observations, the researchers proposed ResQAdam. It maintains only a single BF16-precision momentum, applies Adam updates only to the r×r coordinates of the principal singular subspace, and adds a normalized sign update computed from the residual. Specifically, ResQAdam decomposes the momentum into a principal-subspace part and a residual part: it uses the standard Adam adaptive learning rate for the principal-subspace part, while for the residual part it retains only its sign direction, normalizes it, and adds it to the update. This preserves the useful information in the residual while avoiding the need to store the full momentum state.

Adam's adaptive learning rate and the normalized sign update: Adam maintains first-order momentum (the mean gradient) and second-order momentum (the mean squared gradient) for each parameter, and its update is approximately m/√v—that is, the first-order momentum is normalized by the second-order momentum so that parameters of different magnitudes receive roughly comparable step sizes. ResQAdam follows this mechanism for the r×r coordinates of the principal subspace; for the residual part, since it cannot afford the cost of storing a full second-order momentum, it instead takes only the direction (sign) of the residual, normalizes it, and adds it to the principal-subspace update. This amounts to approximating Adam's normalization effect with a "signSGD-style" sign update—sign updates are inherently insensitive to gradient scale, which is intuitively consistent with Adam's scale normalization, so valid information in the residual can be preserved with almost no extra state.

Experimental Evaluation: Significant Memory Savings, Accuracy Close to AdamW

The authors evaluated the method using two from-scratch QAT methods and three QAT fine-tuning pipelines, quantizing the weights to 2 bits. The results show that ResQAdam reduces optimizer state memory by 71% to 74% relative to AdamW, while achieving accuracy close to AdamW in most settings, with training throughput dropping by less than 2%. This means ResQAdam substantially lowers memory requirements while barely affecting training speed or model quality.

Conclusion and Outlook

Through its innovative design of preserving residual sign information, ResQAdam achieves QAT performance close to AdamW within a low-rank optimization framework while greatly reducing memory usage. This is critical for training low-bit large language models under limited hardware resources. In the future, the method is expected to be extended to other memory-sensitive training scenarios.

One-line Summary

ResQAdam uses a single BF16 momentum plus residual sign updates to cut QAT optimizer memory by about 70% while keeping accuracy close to AdamW.

Paragraph Summary

Quantization-aware training (QAT) is the most reliable way to obtain high-accuracy low-bit LLMs, but AdamW must maintain two FP32 momentum states per weight, creating huge memory overhead. Low-rank optimizers (such as GaLore and LDAdam) can compress states but show notable accuracy degradation in QAT, because a considerable portion of the QAT momentum falls outside the top-r singular subspace and is progressively discarded by projection methods. The paper finds that even retaining only the sign of this residual remains useful, and on this basis proposes ResQAdam: it maintains only a single BF16 momentum, performs Adam updates on the r×r coordinates of the principal subspace, and adds a normalized sign update computed from the residual. Experiments with 2-bit quantization, two from-scratch QAT methods, and three fine-tuning pipelines show that optimizer state memory is reduced by 71%–74% relative to AdamW, accuracy is close to AdamW in most settings, and throughput drops by less than 2%.