CodeQuant2: A New Breakthrough in Low-Precision Quantization for Reasoning Models, Achieving Up to 2.85x Inference Speedup

Large reasoning models (LRMs) improve task performance by generating longer intermediate reasoning traces, but this significantly increases inference costs, making low-precision quantization increasingly important. However, aggressively quantized models often underperform on reasoning tasks. Research finds that larger activation fluctuations occur in the initial decoding steps of the reasoning process, and calibration methods that ignore these early states may miss critical activation fluctuations. To address these reasoning-specific quantization challenges, researchers propose CodeQuant2—a clustering-based lookup table (LUT) quantization framework designed specifically for low-precision large reasoning models.

The Quantization Dilemma of Reasoning Models

Large reasoning models excel at complex tasks, but at the cost of generating lengthy reasoning chains, which significantly drives up inference costs. Low-precision quantization is a key technology for reducing deployment costs, but existing methods often fall short on reasoning tasks. The core issue is that reasoning models produce severe activation fluctuations in the early decoding stage, and traditional calibration processes typically ignore these early states, leading to accumulated quantization errors. CodeQuant2 is proposed precisely to address these unique challenges in the reasoning scenario.

Activation Outliers refer to the phenomenon where, during forward propagation in neural networks, the numerical distributions output by certain layers exhibit extreme large values or severe fluctuations. In large language models, activation values typically follow a long-tail distribution, with a few channels having values far larger than the rest. For reasoning models, activation fluctuations in the early decoding stage are particularly pronounced because when the model first begins generating the reasoning chain, the context is not yet sufficient, and the attention mechanism needs to switch rapidly among multiple candidate directions, causing dramatic changes in hidden states. If the quantization process uses a fixed scaling factor to cover the entire activation range, these outliers will stretch the quantization step size, compressing values within the normal range into a few quantization levels and causing precision loss. Therefore, identifying and handling activation fluctuations in the early decoding stage is a key difficulty in low-bit quantization of reasoning models.

Core Technologies of CodeQuant2

CodeQuant2 adopts a clustering-based LUT quantization framework that integrates several innovative technologies:

• Rotation and scaling to suppress outliers: Rotation and scaling operations mitigate the impact of activation outliers, making weight distributions more amenable to quantization.
• Clustering to represent non-uniform weights: Clustering methods are used to represent non-uniform weight distributions, capturing weight characteristics more accurately.
• Adaptive codebook size: The codebook size is dynamically adjusted according to the quantization difficulty of each layer, achieving optimal resource allocation.

These technologies work together to significantly reduce bit width while maintaining model accuracy.

Look-Up Table (LUT) quantization is an implementation of non-uniform quantization. Traditional linear quantization maps floating-point numbers to integer grid points using a uniform step size, whereas LUT quantization pre-stores a "codebook" for each weight or activation block, where each entry in the table corresponds to a learnable quantization center value. During quantization, one only needs to look up the table to find the nearest center index, and during dequantization, the index is used to retrieve the center value. This approach can flexibly approximate arbitrary distributions and is particularly suitable for scenarios with non-uniform weight distributions or outliers. The cost is the additional storage overhead for saving codebooks and the need for custom kernels to efficiently perform table lookups during inference. The adaptive codebook size in CodeQuant2 is precisely the dynamic adjustment of the number of codebook entries per layer, balancing accuracy and storage.

Improvements in Calibration Strategy

Based on observations of activation behavior during inference, CodeQuant2 improves the calibration sample selection strategy to better match actual decoding statistics. Traditional calibration often uses general text, but the activation patterns of reasoning models fluctuate more in the early decoding stage, so samples that can represent the reasoning process need to be selected. This improvement ensures the stability of the quantized model on reasoning tasks.

Calibration is a key step in post-training quantization (PTQ): a small batch of representative samples is fed into the model to collect statistics on the dynamic range or distribution of activations at each layer, thereby determining quantization parameters (such as scaling factors, zero points, or codebook centers). The distribution of calibration samples directly affects the quality of quantization parameters—if the calibration data distribution does not match the actual inference data distribution, quantization errors will increase significantly. For reasoning models, the conventional practice is to calibrate with general text, but during actual inference the model generates a large number of intermediate reasoning steps, and the activation statistics differ markedly from those of general text. CodeQuant2 emphasizes selecting calibration samples that cover the fluctuation characteristics of the early decoding stage, making quantization parameters closer to the activation distribution during real inference and thus reducing precision loss in the early decoding stage.

Performance

Combined with a custom LUT CUDA kernel, CodeQuant2 achieves significant accuracy improvements on reasoning benchmarks while obtaining up to 2.85x speedup compared to BF16. This result demonstrates that quantization methods designed for the characteristics of reasoning models can effectively balance accuracy and efficiency, providing a viable solution for low-precision deployment of large reasoning models.

Significance and Outlook

The emergence of CodeQuant2 provides new ideas for the deployment of low-precision large reasoning models. Its calibration strategy targeting activation fluctuations during the reasoning process and adaptive clustering method are expected to further promote the application of quantization technology in reasoning scenarios. As the scale of reasoning models continues to grow, such specialized quantization technologies will become increasingly critical.

One-line Summary

CodeQuant2 uses clustering-based LUT quantization and an improved calibration strategy to address early activation fluctuations in reasoning models, achieving up to 2.85x speedup.

Paragraph Summary

Large reasoning models improve performance by generating lengthy reasoning chains, but this significantly increases inference costs, making low-precision quantization critical. However, aggressively quantized models underperform on reasoning tasks, primarily because activation fluctuations in the early decoding stage are ignored by traditional calibration. To this end, researchers propose CodeQuant2—a clustering-based lookup table (LUT) quantization framework. It suppresses outliers through rotation and scaling, uses clustering to represent non-uniform weights, adaptively adjusts codebook size per layer, and improves calibration sample selection to match activation statistics during inference. Combined with a custom LUT CUDA kernel, CodeQuant2 significantly improves accuracy on reasoning benchmarks and achieves up to 2.85x speedup compared to BF16, providing a viable solution for low-precision deployment of large reasoning models.