FP4
A numerical precision format that stores floating-point numbers using 4 bits. Compared with FP16 or BF16, it achieves a fourfold reduction in memory usage at the cost of reduced numerical range and precision. It is commonly used for large model inference optimization.
Core Facts
Timeline (last 90 days)
The precision loss of FP4 is more severe for data such as the KV cache that is sensitive to long-range dependencies, and it usually requires per-channel or per-token scaling strategies, as well as possibly keeping sensitive layers in higher precision.
INT4 uses uniform quantization, while FP4 retains exponent bits, which can better cover the long-tail distribution in activation values.
FP4 quantization usually needs to be used together with block-wise scaling or microscaling factors.
DeepInfer sets three expectations for hardware: expert weights in FP4, activations and KV in FP8, and scaling factors in the low-precision UE8M0 format; KV Cache per token under 1KB; and the bottleneck shifting to large-scale high-speed interconnect and device-side communication.
DeepSeek 1017's expert weights are already FP4 (4 bits per parameter), so conventional 4-bit quantization is of little help; the Engram memory table can be placed on SSD (reading only about 12KB per token), but the expert weights must reside in high-speed memory, because the router selects 6 of 384 experts per layer and makes a different selection for each token.
The RTX 4070 Ti SUPER uses the Ada architecture, has 16GB of VRAM, and has no FP4 compute units.
Low-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
The bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.
An arXiv paper (arXiv:2610.00053) proposes a method called 'format-aware fusion' to optimize the FP4 pretraining process.
The paper finds that the ranking of downstream tasks is not consistent with the ranking of training loss, and schemes with lower training loss do not necessarily perform better in downstream evaluations.
12 more timeline events
All Facts (20)
The NVIDIA Blackwell architecture (GB200/B200 series) introduces native FP4 tensor cores at the hardware level
75%UnverifiedGPT-OSS原生支持FP4混合精度，无需额外量化即可大幅压缩显存占用
90%UnverifiedMoRI（Mixture of Routed Inference）量化通信策略在 dispatch 阶段使用 FP4 精度，combine 阶段使用 FP8 精度，实现了 2.56 倍的带宽压缩
90%UnverifiedBlackwell架构支持FP4精度计算，FP4是目前商用AI芯片支持的最低精度格式
80%Unverified许多深度学习模型在权重量化到4位后仍能保持95%以上的推理精度
75%Unverifiedv4.1 Flash's expert weights are shipped natively in FP4 (4 bits per parameter) and no 8-bit version has ever been released; being close to the compression limit of this format, it cannot be further quantized for Agentic tasks.
60%UnverifiedThe precision loss of FP4 is more severe for data such as the KV cache that is sensitive to long-range dependencies, and it usually requires per-channel or per-token scaling strategies, as well as possibly keeping sensitive layers in higher precision.
50%UnverifiedFP4 quantization usually needs to be used together with block-wise scaling or microscaling factors.
50%UnverifiedINT4 uses uniform quantization, while FP4 retains exponent bits, which can better cover the long-tail distribution in activation values.
50%UnverifiedDeepInfer sets three expectations for hardware: expert weights in FP4, activations and KV in FP8, and scaling factors in the low-precision UE8M0 format; KV Cache per token under 1KB; and the bottleneck shifting to large-scale high-speed interconnect and device-side communication.
50%UnverifiedDeepSeek 1017's expert weights are already FP4 (4 bits per parameter), so conventional 4-bit quantization is of little help; the Engram memory table can be placed on SSD (reading only about 12KB per token), but the expert weights must reside in high-speed memory, because the router selects 6 of 384 experts per layer and makes a different selection for each token.
50%UnverifiedThe RTX 4070 Ti SUPER uses the Ada architecture, has 16GB of VRAM, and has no FP4 compute units.
50%UnverifiedLow-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
50%UnverifiedAn arXiv paper (arXiv:2610.00053) proposes a method called 'format-aware fusion' to optimize the FP4 pretraining process.
50%UnverifiedThe paper finds that the ranking of downstream tasks is not consistent with the ranking of training loss, and schemes with lower training loss do not necessarily perform better in downstream evaluations.
50%UnverifiedValidation of FP4 schemes must establish evaluation benchmarks covering diverse downstream tasks, and convergence to a lower loss cannot be the sole criterion for scheme selection.
50%UnverifiedThe success or failure of FP4 training is jointly determined by the scaling contract, operand format, and execution path—it is a systemic engineering problem rather than merely a computational precision problem.
50%UnverifiedThe bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.
50%UnverifiedThe standard FP4 format can only represent a dozen or so distinct values and must be paired with a scale factor mechanism to cover the distribution range of actual model weights.
50%UnverifiedThe DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.
50%