INT4

4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.

INT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.

Quantization compresses the storage precision of model parameters from 16-bit down to 8-bit or even 4-bit, thereby reducing memory usage, but at the cost of potential loss in model accuracy.

The core principle of quantization is to convert model parameters from high-precision floating-point representations (such as FP32's 32-bit floating point) to low-precision representations (such as INT8's 8-bit integer or INT4's 4-bit integer).

Re-quantizing INT4 weights to NF4 subjects each weight to two rounds of rounding, and official measurements show this increases the perplexity of Qwen3-8B by 1.6%

INT4 uses uniform quantization, while FP4 retains exponent bits, which can better cover the long-tail distribution in activation values.

NVFP4, with its finer block granularity and floating-point scaling, better preserves small-value information in layers where activation distributions span a wide range, so at the same 4-bit width it often incurs less perplexity loss than INT4.

INT4 is fixed-point integer quantization with limited dynamic range.

For uniform quantization, the theoretical SQNR ≈ 6.02 × bit count + 1.76 dB, so the theoretical upper limit is about 49.9 dB for INT8 and about 25.8 dB for INT4.

Applying a block-diagonal Hadamard transform to the INT4 cache brings about a 0.20 increase in perplexity but in exchange reduces cache quantization error.

FuTransformer is a CUDA implementation targeting NVIDIA Hopper GPUs, with a separate FP8 prefill path and INT4 decode path designed for Llama-style Transformer blocks.

The A100 is also Ampere architecture, so running FP8-quantized models on the A100 is also inefficient, and an INT4 quantization scheme should be chosen.

Commonly used inference frameworks include vLLM, TensorRT-LLM, Ollama, etc., while quantization commonly uses methods such as INT8 and INT4 to compress model size.

A common industry practice is to quantize weights to INT4 or INT8 while retaining higher precision for critical layers, in order to strike a balance between cost and quality.

Unsloth's Triton kernels dequantize layer by layer on demand, keeping INT4 precise without re-quantizing to NF4, avoiding an approximately 1.6% increase in perplexity.

Quantization precision recommendations: prioritize BF16 for production environments, INT4 is acceptable for test environments, and the minimum threshold should not go below INT4, as lower precision results in greater loss of model capability and output quality.

The difference between NVFP4 and INT4 is that NVFP4 retains the dynamic range expressiveness of floating-point numbers, making it more friendly to large models with non-uniform weight distributions.

Under FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.

When using INT4 HQQ quantization, catastrophic transcription loops occurring when the model processes West African accents were amplified by 5 to 7 times.

In edge device and low-precision inference scenarios (such as INT8/INT4 quantization), exponential operations are often a concentrated source of quantization error and computational overhead