FP8

NVFP4 is a 4-bit floating-point (FP4) quantization format introduced by NVIDIA, designed specifically for the hardware acceleration capabilities of Blackwell architecture GPUs

以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB

FP8 quantization technology reduces memory usage by approximately 50% with almost no loss in precision by compressing weights into an 8-bit representation.

FP8 is an 8-bit floating-point format natively supported by newer architectures such as the NVIDIA H100, which can compress data volume by half compared to FP16/BF16

Quantization (such as INT8 and FP8) can reduce memory usage and improve throughput while maintaining acceptable accuracy loss in most scenarios.

FP8 data volume is half that of FP16, and the H100's FP8 Tensor Core compute throughput is twice that of FP16 (approximately 1979 TFLOPS).

Even compressing the weights to 214GB using 4-bit quantization still far exceeds the capacity of a single 48GB GPU.

The model weights of Flux 2's 9B parameter model occupy approximately 18GB of VRAM at FP16 precision, typically requiring more than 24GB of GPU VRAM.

MoRI（Mixture of Routed Inference）量化通信策略在 dispatch 阶段使用 FP4 精度，combine 阶段使用 FP8 精度，实现了 2.56 倍的带宽压缩

NVFP4 occupies 4 bits per parameter, i.e., 0.5 bytes, theoretically halving weight memory usage and load bandwidth compared with FP8.

On a dual-Spark system, DeepSeek V4 Flash (FP8 quantization) peaks at approximately 53 tokens per second.

FP8 is usually divided into two variants, E4M3 and E5M2; the former has higher precision and the latter a larger dynamic range, and during inference they can be paired with scaling factors to improve throughput.

The weights of Nemotron 3.5 Transcribe are released in three precisions: BF16, FP8, and NVFP4.

In MoE inference, lowering weight bit width from FP16 to FP8/INT8 reduces the number of bytes proportionally, thereby shortening latency, provided that model throughput is limited by bandwidth rather than compute.

The optimization measures for running Hy-Nomas on a single RTX 4090 include an INT8-quantized DiT model, an FP8 version of the video decoder, and a CPU offload strategy.

DeepInfer sets three expectations for hardware: expert weights in FP4, activations and KV in FP8, and scaling factors in the low-precision UE8M0 format; KV Cache per token under 1KB; and the bottleneck shifting to large-scale high-speed interconnect and device-side communication.

DeepSeek already proposed during the V3.1 period that UE8M0 FP8 was designed for next-generation domestic chips; this time it is fully implemented in DeepInfer, meaning chips that do not support this format will directly suffer in precision or performance.

In FP8 mixed-precision training, V3 uses FP8 computation for most matrix multiplications and retains higher precision for precision-sensitive steps, theoretically doubling compute throughput and halving memory usage compared with BF16.

The baseline BF16 model reaches about 37.5 tokens/s at single concurrency, 714 tokens/s aggregate throughput at 32 concurrent requests, and 61ms first-token latency; after switching to FP8 weights, single concurrency rises to 53.4 tokens/s, and peak throughput at 64 concurrency rises from about 800 to 1,140 tokens/s, about 1.42x.

After FP8 quantization, first-token latency rises noticeably with concurrency: 277ms at 32 concurrent requests, jumping to 1,300ms at 64; beyond 32 concurrent requests, throughput stops growing and latency deteriorates sharply.