BF16

BF16 was originally designed by the Google Brain team for TPUs, retaining the same 8-bit exponent as FP32 while reducing the mantissa from 23 bits to 7 bits.

A 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.

BF16 (Brain Floating Point 16) is a floating-point variant proposed by Google that retains the 8-bit exponent range of FP32 but reduces the mantissa to 7 bits.

Compared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.

The core principle of quantization is to convert model parameters from high-precision floating-point representations (such as FP32's 32-bit floating point) to low-precision representations (such as INT8's 8-bit integer or INT4's 4-bit integer).

BF16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits, sacrificing precision while retaining the same dynamic range as FP32.

混合精度训练将权重保存为FP32，而前向和反向传播计算使用FP16或BF16，可将显存占用减少近一半

The 816B-parameter Muse Spark Open occupies about 493GB of memory after NVFP4 quantization, while FP16 storage would require more than 1.6TB.

BF16 is a 16-bit floating-point format introduced by Google that has a wider exponent range compared to FP16, making it less prone to numerical overflow.

The weights of Nemotron 3.5 Transcribe are released in three precisions: BF16, FP8, and NVFP4.

The study conducted experiments on four Qwen series models, and the results show that simultaneously aligning the three regions of RoPE, SwiGLU, and residual RMSNorm can completely prevent training collapse under the default BF16 baseline.

BF16 (Brain Floating Point 16) is a 16-bit floating-point format that retains an 8-bit exponent (the same as FP32) but has only a 7-bit mantissa, thus offering a large dynamic range but low precision.

A recent study proposes that by aligning the rounding positions in inference and training, BF16 reinforcement learning training can be stabilized without changing the inference engine, matrix multiplication and attention kernels, or learning objectives.

The method proposed in the study keeps selected intermediate results at FP32 precision in three key regions—RoPE (rotary position embedding), SwiGLU (activation function), and residual RMSNorm—and only rounds at their BF16 output stage.

Under BF16, if the result of any floating-point operation cannot be exactly represented in BF16, it must be rounded to the nearest representable value, producing a relative error on the order of about 2^-8.

n-gram tables are usually stored at full precision (such as FP16 or BF16), further amplifying memory usage.

The baseline BF16 model reaches about 37.5 tokens/s at single concurrency, 714 tokens/s aggregate throughput at 32 concurrent requests, and 61ms first-token latency; after switching to FP8 weights, single concurrency rises to 53.4 tokens/s, and peak throughput at 64 concurrency rises from about 800 to 1,140 tokens/s, about 1.42x.

The weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.

At BF16 precision, each model parameter requires 2 bytes of storage.

In this test, NVFP4 achieved a generation speed of 116.8 tok/s, 2.3 times that of BF16 (51.0 tok/s), and the NVFP4 weight size was 3.28 GiB.