GPTQ
A post-training quantization method proposed in 2022 that uses layer-wise second-order optimization to compress large models to INT4 precision, representing a significant milestone in large language model quantization.
Core Facts
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
Common model quantization methods include formats such as GPTQ, AWQ, and GGUF.
4-bit quantization techniques (such as GPTQ, AWQ, and the GGUF format) can reduce model VRAM requirements to 1/4 of the original, but come with a certain degree of accuracy loss
Timeline (last 90 days)
Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 (round-to-nearest weights) or 6.99 (GPTQ in the same format).
Different quantization formats (GGUF's Q4_K_M, AWQ, GPTQ, MLX's 4bit, etc.) differ in speed and quality.
Both AWQ and GPTQ require calibration data and corresponding operator support on the inference side, making them less directly usable than GGUF in a pure browser WASM environment.
Different quantization formats (GGUF, AWQ, GPTQ, bitsandbytes) have different metadata overhead and layouts, and peak memory during loading may also differ.
4-bit quantization (such as bitsandbytes NF4 or GPTQ) weights are usually stored in blocks, with each block carrying a scaling factor and zero point, and these metadata are often stored in FP16 or FP32; embedding layers, output layers, and normalization layers may remain at higher precision, so the actual weight file is typically 5%-15% larger than the theoretical P×b/8 value.
1.5 bpw is already close to the extremely low range in the information-theoretic sense, and usually requires methods such as GPTQ, AWQ, or vector quantization, along with a calibration dataset, to suppress accuracy collapse.
Running a 137B model locally usually adopts a hybrid quantization strategy, such as keeping higher precision for key layers and using 4-bit or 3-bit quantization for secondary experts, or adopting formats such as GGUF, AWQ, and GPTQ with inference frameworks such as llama.cpp and vLLM.
Mainstream quantization schemes include GPTQ, AWQ, GGUF, and others, which perform group quantization according to the distribution characteristics of the weights and can achieve a good balance between compression rate and fidelity.
The actual effective bits per weight (bpw) of mainstream quantization formats (such as AWQ, GPTQ, and GGUF K-Quants) is usually higher than 4; for example, INT4 AWQ with group_size=128 is about 4.5 bpw, and GGUF Q4_K_M is about 4.85 bpw.
A 125-billion-parameter model can run on 12G of VRAM, relying mainly on model quantization and a mixture-of-experts (MoE) architecture, achieved by activating only part of the expert parameters, quantizing weights to 4bit or lower, and dynamically loading inactive parameters from memory or SSD on demand.
40 more timeline events
All Facts (20)
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%Verified4-bit quantization techniques (such as GPTQ, AWQ, and the GGUF format) can reduce model VRAM requirements to 1/4 of the original, but come with a certain degree of accuracy loss
90%VerifiedAWQ (2023) found that approximately 1% of salient channels in the weights have the greatest impact on model output, and protects these channels through equivalent scaling.
85%Verified4-bit quantization reduces model size to 1/8 of the original at the cost of approximately 1–3% accuracy loss.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedThe mainstream quantization formats in the community include GGUF, AWQ, and GPTQ, among which GGUF was popularized by the llama.cpp project and supports CPU inference.
80%VerifiedAWQ identifies and protects the weights with the greatest influence on activation values, and typically achieves better accuracy than GPTQ at the same compression ratio.
80%VerifiedThe core insight of AWQ is that only approximately 1% of salient weights have a major impact on model output, and protecting them enables model quality to be maintained at extremely low bit widths
75%VerifiedINT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.
75%VerifiedCommon quantization schemes include GPTQ (requires GPU), AWQ, and GGUF format quantization within the llama.cpp ecosystem.
65%VerifiedInference frameworks such as llama.cpp, Ollama, and vLLM are compatible with Qwen model weights and support quantization formats such as GGUF, AWQ, and GPTQ, enabling them to run on consumer-grade GPUs and even CPUs.
65%VerifiedModern quantization schemes such as GPTQ and AWQ introduce layer-by-layer calibration and error compensation mechanisms to keep perplexity loss within 1–2%
65%VerifiedStacking techniques such as K-quant, GPTQ, and AWQ can smoothly extend quantization to 4-bit, but performance catastrophically collapses when pushed to 2-bit or 1-bit.
65%VerifiedOllama uses the GGUF format by default; other common quantization formats include GPTQ.
65%VerifiedQuantization compression techniques such as GPTQ and AWQ can compress model size to 1/4 to 1/8 of the original with almost no loss of accuracy through layer-by-layer error compensation.
65%VerifiedModern quantization schemes such as GPTQ, AWQ, and the grouped quantization of the GGUF format can reduce model size by more than 75% while retaining the vast majority of reasoning capability through layer-by-layer calibration and outlier protection.
65%VerifiedGPTQ is a post-training quantization scheme based on layer-wise error compensation, with minimal precision loss.
65%VerifiedMainstream quantization schemes include GPTQ, AWQ, and GGML's Q4_K_M, each making different trade-offs between compression ratio and precision loss.
65%