Q4_K_M

In GGUF quantization naming, Q4_K_M denotes a 4-bit quantization, Medium-precision variant using the K-Quant grouped quantization method

The Qwen2.5:7B model is approximately 4 GB in size after Q4 quantization.

Q4_K_M中的K代表k-quants方案，M代表中等质量级别，是llama.cpp引入的混合量化策略

Quantization is a model compression technique; Q4 is easier to run, while Q8 retains higher quality but consumes more memory

Community rule of thumb holds that Q4_K_M (4-bit medium-precision quantization) offers the best balance between performance and file size in most scenarios.

GGUF格式中的Q4_K_M采用分组量化（Group Quantization）策略，每组独立计算缩放因子和零点

The perplexity loss of Q4_K_M quantization compared to FP16 is typically within 1%–3%

Different quantization types (such as Q4_K_M and Q8_0) each have trade-offs in size, speed and quality.

Q8_0 is 8-bit block quantization with high precision but limited compression; Q4_K_M balances size and quality through mixed precision at the 4-bit level and is one of the commonly used default tiers for local deployment.

Llama.cpp's llama-quantize tool implements block-wise quantization; the K series in the naming (such as Q4_K_M) denotes the k-quants mixed quantization strategy, and M stands for the medium tier.

Across 12 blocks and 1282 scored positions, when Q4_K_M recovers BF16, attention eliminates 0.030 (95% CI 0.019–0.041) and MLP eliminates 0.036 (0.024–0.048), with the paired difference crossing zero so the two are indistinguishable; when injecting Q4_K_M into BF16, MLP adds more (0.014, 0.005 to 0.023).

With flash attention disabled, Q4_K_M outputs are bitwise identical at ubatch sizes that are multiples of 4 and not at other sizes, indicating that llama.cpp's CPU path is not batch-invariant across all batch sizes.

At the is_prime function definition, the BF16 version has a 97.9% probability of outputting False, whereas Q4_K_M has a 79.1% probability of outputting True.

In a comparison of Qwen3-0.6B in BF16 versus Q4_K_M, Q4_K_M changed the top-1 token at 11.6% of positions, with an average KL divergence of 0.067 and perplexity rising from 7.83 to 8.53.

At the is_prime token, after writing BF16's residual into Q4_K_M, that token's KL dropped from 0.48 after layer 6 to 0.04 after layer 9; the reverse write reproduced the flip from about layer 12, indicating the decisive difference forms between layers 6 and 12.

Layer-by-layer denoising experiments between Q4_K_M and BF16 show KL decreasing smoothly from layer 0 to layer 27, with no single layer carrying all the difference.

Q4_K_M量化大约表示每个权重4位，并对异常值采用特定方案，在内存受限的机器上通常是甜点选择。

The comparison baseline Q4_K_M GGUF version runs at about 42 tok/s on llama.cpp on the same card (RTX 3090); HyperQwen is about three times faster in real chat and about six times faster in citation scenarios.

W4A16 quantisation refers to a mixed-precision scheme in which weights are stored in 4-bit and activations remain 16-bit; compared with Q4_K_M it retains a higher-precision activation path, and combined with dedicated CUDA kernels such as marlin it can achieve higher throughput on NVIDIA graphics cards, but VRAM usage is more rigid and kernel support requirements are stricter.

Q4_K_M is a GGUF quantization format widely used in the llama.cpp community, in which weights are compressed to about 4 bits using the medium tier of the k-quant mixed quantization scheme, striking a compromise between size and precision.