Inductor
The backend component of the PyTorch 2.0 compilation stack, responsible for lowering computation graphs into efficient Triton GPU kernels or C++ code for CPU execution. It constitutes the final execution layer through which torch.compile achieves performance acceleration.
Timeline (last 90 days)
The Inductor component of the torch-spyre compilation stack is responsible for deciding how each kernel's work is partitioned across Spyre chip cores and for managing the residency strategy of tensors between on-chip scratchpad and off-chip memory.
The design of the torch-spyre compilation stack draws on Inductor's experience integrating with other accelerators, but it makes special handling for Spyre's explicit memory hierarchy.
By implementing Inductor's pluggable backend interface, torch-spyre reuses Inductor's scheduling capabilities on Spyre, thereby allowing PyTorch models to run on the chip without hand-written kernels.
Inductor is the default compilation backend of torch.compile in PyTorch 2.x, responsible for splitting FX graphs into kernels, generating Triton or C++ code, and providing a pluggable backend interface for different accelerators.
In its adaptation for Spyre, IBM did Spyre-specific scheduling and memory hierarchy adaptation on top of Inductor, needing only to map the intermediate representation output by Inductor to the Spyre instruction set.
The IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.
Vendors such as Intel and AMD also adopt the strategy of plugging into the Inductor extension mechanism and doing hardware-specific optimizations through custom backend interfaces, rather than maintaining parallel compilation pipelines.
When generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
Inductor is becoming the de facto intermediate layer of the AI chip ecosystem, reducing the complexity of chip-to-framework adaptation from O(M×N) to close to O(M+N).
With the continuous investment of Microsoft, Intel, AMD and IBM, the quality of Inductor's general-purpose code generation on non-NVIDIA hardware can already cover a considerable portion of training and inference workloads.
7 more timeline events
All Facts (16)
The Inductor component of the torch-spyre compilation stack is responsible for deciding how each kernel's work is partitioned across Spyre chip cores and for managing the residency strategy of tensors between on-chip scratchpad and off-chip memory.
50%UnverifiedThe design of the torch-spyre compilation stack draws on Inductor's experience integrating with other accelerators, but it makes special handling for Spyre's explicit memory hierarchy.
50%UnverifiedBy implementing Inductor's pluggable backend interface, torch-spyre reuses Inductor's scheduling capabilities on Spyre, thereby allowing PyTorch models to run on the chip without hand-written kernels.
50%UnverifiedInductor is the default compilation backend of torch.compile in PyTorch 2.x, responsible for splitting FX graphs into kernels, generating Triton or C++ code, and providing a pluggable backend interface for different accelerators.
50%UnverifiedIn its adaptation for Spyre, IBM did Spyre-specific scheduling and memory hierarchy adaptation on top of Inductor, needing only to map the intermediate representation output by Inductor to the Spyre instruction set.
50%UnverifiedThe IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.
50%UnverifiedVendors such as Intel and AMD also adopt the strategy of plugging into the Inductor extension mechanism and doing hardware-specific optimizations through custom backend interfaces, rather than maintaining parallel compilation pipelines.
50%UnverifiedWhen generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
50%UnverifiedInductor is becoming the de facto intermediate layer of the AI chip ecosystem, reducing the complexity of chip-to-framework adaptation from O(M×N) to close to O(M+N).
50%UnverifiedAs a general-purpose backend, in workloads that are extremely latency-sensitive or highly customized at the operator level, Inductor's general passes may be inferior to specialized compilers, and IBM needs to retain a handwritten optimization channel for Spyre's key operators.
50%UnverifiedInductor iterates quickly, and its interfaces and internal IR change from time to time, so hardware vendors' adaptation layers need to keep following up, which constitutes a kind of maintenance cost.
50%UnverifiedTorchInductor is PyTorch's default backend compiler, capable of lowering computation graphs into Triton GPU kernels or C++/OpenMP CPU code.
50%UnverifiedIn PyTorch 2.14, NVGEMM brings CUTLASS kernels generated by CuTeDSL into the Inductor compilation backend.
50%UnverifiedInductor is the default compilation backend for PyTorch 2.x, responsible for lowering computation graphs into efficient low-level code such as Triton or C++.
50%UnverifiedThe Inductor backend lowers computation graphs to efficient Triton GPU kernels or C++ code for CPU.
50%UnverifiedBringing CUTLASS kernels into Inductor is expected to deliver substantial performance gains for GEMM-intensive workloads such as Transformers and large language model training and inference.
50%