Kernel Fusion
An optimization technique commonly used in deep learning compilers and inference engines that merges multiple small operators into a single large kernel to reduce kernel launch overhead and memory access, thereby improving GPU utilization.
Core Facts
Timeline (last 90 days)
If steps such as descriptor computation, normalization and nonlinear activation are each made into separate kernels, repeatedly writing intermediate results to global memory causes bandwidth bottlenecks; operator fusion can keep intermediate tensors in on-chip memory.
Kernel fusion is a common optimization used by deep learning compilers and inference engines, merging multiple small operators into one large kernel to reduce launch overhead and memory round trips.
Kernel fusion techniques can merge multiple consecutive operations into a single kernel, for example fusing RoPE with attention computation or fusing SwiGLU's matrix multiplication with activation, thereby reducing memory round trips and scheduling overhead.
This time vLLM reduced the compute and memory access overhead of the prefill stage through means such as SWA bounded replay and kernel fusion.
Kernel fusion can reduce kernel launch overhead and avoid repeated reads and writes of intermediate results to global memory, thereby improving performance.
All Facts (5)
Kernel fusion can reduce kernel launch overhead and avoid repeated reads and writes of intermediate results to global memory, thereby improving performance.
75%UnverifiedIf steps such as descriptor computation, normalization and nonlinear activation are each made into separate kernels, repeatedly writing intermediate results to global memory causes bandwidth bottlenecks; operator fusion can keep intermediate tensors in on-chip memory.
50%UnverifiedKernel fusion is a common optimization used by deep learning compilers and inference engines, merging multiple small operators into one large kernel to reduce launch overhead and memory round trips.
50%UnverifiedKernel fusion techniques can merge multiple consecutive operations into a single kernel, for example fusing RoPE with attention computation or fusing SwiGLU's matrix multiplication with activation, thereby reducing memory round trips and scheduling overhead.
50%UnverifiedThis time vLLM reduced the compute and memory access overhead of the prefill stage through means such as SWA bounded replay and kernel fusion.
50%