AWQ
A weight quantization technique that analyzes activation value distributions to protect critical weights, achieving lower precision loss compared to other quantization approaches at equivalent bit widths.
Core Facts
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
NVFP4 is a 4-bit floating-point (FP4) quantization format introduced by NVIDIA, designed specifically for the hardware acceleration capabilities of Blackwell architecture GPUs
AWQ (Activation-aware Weight Quantization) technology identifies critical weight channels for special protection by observing activation values during model inference, and compresses the remaining weights to 4-bit precision.
Timeline (last 90 days)
Different quantization formats (GGUF's Q4_K_M, AWQ, GPTQ, MLX's 4bit, etc.) differ in speed and quality.
AWQ is a 4-bit weight quantization method that preserves accuracy by protecting the weight channels that have a large impact on activations, reducing memory usage while still requiring dequantization during computation.
With the 7B AWQ model at batch 32 the 3090 already lags behind the L40S (0.71x), whereas the 7B fp16 is not overtaken until batch 128 (0.72x), showing that the quantized model makes the compute bottleneck appear earlier.
Both AWQ and GPTQ require calibration data and corresponding operator support on the inference side, making them less directly usable than GGUF in a pure browser WASM environment.
Different quantization formats (GGUF, AWQ, GPTQ, bitsandbytes) have different metadata overhead and layouts, and peak memory during loading may also differ.
1.5 bpw is already close to the extremely low range in the information-theoretic sense, and usually requires methods such as GPTQ, AWQ, or vector quantization, along with a calibration dataset, to suppress accuracy collapse.
Running a 137B model locally usually adopts a hybrid quantization strategy, such as keeping higher precision for key layers and using 4-bit or 3-bit quantization for secondary experts, or adopting formats such as GGUF, AWQ, and GPTQ with inference frameworks such as llama.cpp and vLLM.
Mainstream quantization schemes include GPTQ, AWQ, GGUF, and others, which perform group quantization according to the distribution characteristics of the weights and can achieve a good balance between compression rate and fidelity.
AWQ (Activation-aware Weight Quantization) is a post-training quantization method that preserves accuracy as much as possible during 4-bit quantization by referencing activation values to determine which channels are more important.
The actual effective bits per weight (bpw) of mainstream quantization formats (such as AWQ, GPTQ, and GGUF K-Quants) is usually higher than 4; for example, INT4 AWQ with group_size=128 is about 4.5 bpw, and GGUF Q4_K_M is about 4.85 bpw.
40 more timeline events
All Facts (20)
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedNVFP4 is a 4-bit floating-point (FP4) quantization format introduced by NVIDIA, designed specifically for the hardware acceleration capabilities of Blackwell architecture GPUs
90%VerifiedAWQ (Activation-aware Weight Quantization) technology identifies critical weight channels for special protection by observing activation values during model inference, and compresses the remaining weights to 4-bit precision.
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%Verified4-bit quantization techniques (such as GPTQ, AWQ, and the GGUF format) can reduce model VRAM requirements to 1/4 of the original, but come with a certain degree of accuracy loss
90%VerifiedAWQ (2023) found that approximately 1% of salient channels in the weights have the greatest impact on model output, and protects these channels through equivalent scaling.
85%Verified4-bit quantization reduces model size to 1/8 of the original at the cost of approximately 1–3% accuracy loss.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedThe mainstream quantization formats in the community include GGUF, AWQ, and GPTQ, among which GGUF was popularized by the llama.cpp project and supports CPU inference.
80%VerifiedAWQ identifies and protects the weights with the greatest influence on activation values, and typically achieves better accuracy than GPTQ at the same compression ratio.
80%VerifiedThe core insight of AWQ is that only approximately 1% of salient weights have a major impact on model output, and protecting them enables model quality to be maintained at extremely low bit widths
75%VerifiedINT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.
75%VerifiedCommon quantization schemes include GPTQ (requires GPU), AWQ, and GGUF format quantization within the llama.cpp ecosystem.
65%VerifiedInference frameworks such as llama.cpp, Ollama, and vLLM are compatible with Qwen model weights and support quantization formats such as GGUF, AWQ, and GPTQ, enabling them to run on consumer-grade GPUs and even CPUs.
65%VerifiedModern quantization schemes such as GPTQ and AWQ introduce layer-by-layer calibration and error compensation mechanisms to keep perplexity loss within 1–2%
65%VerifiedStacking techniques such as K-quant, GPTQ, and AWQ can smoothly extend quantization to 4-bit, but performance catastrophically collapses when pushed to 2-bit or 1-bit.
65%VerifiedOllama uses the GGUF format by default; other common quantization formats include GPTQ.
65%VerifiedQuantization compression techniques such as GPTQ and AWQ can compress model size to 1/4 to 1/8 of the original with almost no loss of accuracy through layer-by-layer error compensation.
65%VerifiedModern quantization schemes such as GPTQ, AWQ, and the grouped quantization of the GGUF format can reduce model size by more than 75% while retaining the vast majority of reasoning capability through layer-by-layer calibration and outlier protection.
65%