GEMM
General Matrix Multiplication, the fundamental and computationally intensive operation in deep learning. It is the core computation underlying fully connected layers, attention mechanism Q/K/V projections, and Transformer feed-forward networks, with its execution efficiency directly determining overall model training and inference throughput.
Core Facts
Timeline (last 90 days)
For dense GEMM and grouped GEMM, PyTorch's scheduling layer selects the appropriate backend based on tensor shapes and hardware type.
FlyDSL covers core operations such as dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm and TopK.
Matrix multiplication satisfies (A×B)ᵀ = Bᵀ×Aᵀ; swapping the two operands and transposing the result does not change the computational semantics, only the way data is arranged in memory and registers.
In Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
In small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
GEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
Benchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
The two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
Floating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
FP8 has a narrower numerical range and lower precision, and correct usage requires fine-grained scaling factor management and numerical stability guarantees at the GEMM kernel level.
7 more timeline events
All Facts (18)
General Matrix Multiplication (GEMM) is the most fundamental and computationally intensive operation in deep learning inference, and NVIDIA Tensor Cores and Google TPU systolic arrays are both specialized optimizations targeting GEMM.
75%Unverified通过CUDA Tile，开发者可以更自然地表达矩阵分块乘法（Tiled GEMM），而不必深入处理wmma或mma.sync等底层PTX指令
85%UnverifiedFor dense GEMM and grouped GEMM, PyTorch's scheduling layer selects the appropriate backend based on tensor shapes and hardware type.
50%UnverifiedFlyDSL covers core operations such as dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm and TopK.
50%UnverifiedMatrix multiplication satisfies (A×B)ᵀ = Bᵀ×Aᵀ; swapping the two operands and transposing the result does not change the computational semantics, only the way data is arranged in memory and registers.
50%UnverifiedFloating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
50%UnverifiedIn small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
50%UnverifiedGEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
50%UnverifiedBenchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
50%UnverifiedThe two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
50%UnverifiedIn Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
50%UnverifiedIn large models based on the Transformer architecture, the vast majority of floating-point operations are concentrated in matrix multiplication, and the efficiency of GEMM kernels directly determines training throughput and inference latency.
50%UnverifiedFP8 has a narrower numerical range and lower precision, and correct usage requires fine-grained scaling factor management and numerical stability guarantees at the GEMM kernel level.
50%UnverifiedThe computational form of GEMM can be described as C = A × B + C, where A, B, and C are all two-dimensional matrices.
50%UnverifiedBLAS (Basic Linear Algebra Subprograms) is the foundational building block of deep learning and scientific computing, with neural network training and inference relying on large-scale matrix multiplication (GEMM).
50%UnverifiedTileLang's most direct application scenarios are concentrated on core deep learning operators such as FlashAttention, GEMM (General Matrix Multiplication), and quantization and dequantization.
50%UnverifiedThe inference bottlenecks of MoE models lie mainly in two areas: KV Cache read/write (constrained by memory bandwidth) and GEMM computation for MoE-routed experts (a compute-intensive task with irregular shapes).
50%UnverifiedThe experiment was conducted on a nine-core FP16 GEMM (half-precision general matrix multiplication) test set.
50%