INT8

Quantization (such as INT8 and FP8) can reduce memory usage and improve throughput while maintaining acceptable accuracy loss in most scenarios.

The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.

The core principle of quantization is to convert model parameters from high-precision floating-point representations (such as FP32's 32-bit floating point) to low-precision representations (such as INT8's 8-bit integer or INT4's 4-bit integer).

In MoE inference, lowering weight bit width from FP16 to FP8/INT8 reduces the number of bytes proportionally, thereby shortening latency, provided that model throughput is limited by bandwidth rather than compute.

The optimization measures for running Hy-Nomas on a single RTX 4090 include an INT8-quantized DiT model, an FP8 version of the video decoder, and a CPU offload strategy.

For uniform quantization, the theoretical SQNR ≈ 6.02 × bit count + 1.76 dB, so the theoretical upper limit is about 49.9 dB for INT8 and about 25.8 dB for INT4.

Quantization techniques can compress trained neural network weights from FP16/FP32 to INT8/INT4, significantly reducing model size, lowering memory usage, and improving inference speed, at the cost of possible accuracy loss.

For existing post-training quantization methods to reach precision close to FP32, they basically must use INT8 fixed-point integers, and performance drops noticeably when the bit width is below INT8.

Commonly used inference frameworks include vLLM, TensorRT-LLM, Ollama, etc., while quantization commonly uses methods such as INT8 and INT4 to compress model size.

A common industry practice is to quantize weights to INT4 or INT8 while retaining higher precision for critical layers, in order to strike a balance between cost and quality.

Integer arithmetic on CPUs is generally faster than floating-point arithmetic, especially when combined with dedicated acceleration instructions, which can improve inference speed and reduce power consumption.

Image diffusion inference introduces INT8/FP8 support, with a speed improvement of approximately 2x.

INT8/FP8 image diffusion inference delivers approximately 2x speed improvement

In edge device and low-precision inference scenarios (such as INT8/INT4 quantization), exponential operations are often a concentrated source of quantization error and computational overhead

Using a quantization combination of INT8 weights with INT4 embedding layers in ONNX, the F1 score difference is only approximately 0.005 compared to more conservatively quantized versions.

Mastering TensorRT optimization typically requires understanding specialized knowledge such as engine building and precision calibration (e.g., INT8/FP8 quantization).

When evaluating quantization schemes, one should look at the number of qualified outputs per unit time rather than simply single-inference speed, as the speed gains from INT4 may be offset by the overhead of repeated regeneration.

The user used only ComfyUI's standard native workflow and replaced the Qwen encoder with the INT8 version, achieving quality improvements without any special workflow configuration.

LTX 2.5 splits model components into separate files and directly provides INT8 and NVFP4 quantization formats.

Quantizing from FP32 to INT8 can reduce the model size to one quarter of the original.