Concept块对角Hadamard变换

Hadamard Transform

An orthogonal linear transform that distributes a vector's energy uniformly across its dimensions. In quantization it is used to spread out the energy of outlier channels and thereby reduce quantization error.

Timeline (last 90 days)

Oct 19

Applying a block-diagonal Hadamard transform to the INT4 cache brings about a 0.20 increase in perplexity but in exchange reduces cache quantization error.

Unverified50%

All Facts (1)

Unverified

Applying a block-diagonal Hadamard transform to the INT4 cache brings about a 0.20 increase in perplexity but in exchange reduces cache quantization error.

50%

Source Articles

FP8 Prefill + INT4 Decode: New Optimization for Llama Inference on HopperOct 19