FP16

In FP16 precision, large models require approximately 2 GB of VRAM per 1 billion parameters.

In mixed-precision training, the forward pass uses FP16 to leverage Tensor Core acceleration while parameter updates retain FP32 to maintain numerical stability, which has become standard practice for large model training.

A 35B model requires approximately 70GB of VRAM at FP16 precision, far exceeding the RTX 5090's 32GB capacity, so single-GPU deployment requires quantization.

Compared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.

The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.

Quantization compresses the storage precision of model parameters from 16-bit down to 8-bit or even 4-bit, thereby reducing memory usage, but at the cost of potential loss in model accuracy.

FP16格式下，大模型显存需求的估算公式为：参数量 × 2 = 所需显存（GB），即每个参数以16位（2字节）浮点数存储

A 7B-parameter model requires approximately 14 GB of VRAM for model weights alone in FP16 format, and the total VRAM requirement for training may balloon to 50–80 GB.

The perplexity loss of Q4_K_M quantization compared to FP16 is typically within 1%–3%

A 7B-parameter model requires about 14GB of VRAM at FP16 precision, and actual requirements are higher once KV Cache and inference framework overhead are added.

In MoE inference, lowering weight bit width from FP16 to FP8/INT8 reduces the number of bytes proportionally, thereby shortening latency, provided that model throughput is limited by bandwidth rather than compute.

n-gram tables are usually stored at full precision (such as FP16 or BF16), further amplifying memory usage.

The system applies per-row range protection to FP16 outputs: before quantizing or writing out FP16, it computes the maximum value of each row and dynamically adjusts the scaling factor accordingly so that the row's values fall within FP16's safe range, then restores them on readout.

FP16 has only 5 exponent bits, and the smallest representable normal number is about 6×10^-5, so input gradients in backward propagation easily underflow to zero, causing shallower layers to learn nothing at all.

Models are often trained with FP32 or mixed precision, and are usually converted to FP16 when releasing inference versions to halve the size.

FP16 is the default precision for most models at release.

Quantization techniques can compress trained neural network weights from FP16/FP32 to INT8/INT4, significantly reducing model size, lowering memory usage, and improving inference speed, at the cost of possible accuracy loss.

W8A8 quantization means both weights and activations are compressed to 8-bit integers; compared with FP16 it can reduce memory usage and improve NPU throughput, at the cost of possibly introducing slight accuracy loss.

Low-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.

Compared with FP16, 4-bit quantisation can reduce model size and memory usage to about a quarter, which is the key prerequisite for ordinary CPU machines to run local large models.