torch.compile

TorchDynamo is the entry layer of the PyTorch 2.0 compilation stack, converting dynamic graph code into optimizable intermediate representations through Python bytecode interception.

Mixed-precision training can typically increase training speed by 1.5–2 times and reduce VRAM usage by nearly half.

torch.compile is a core compilation feature introduced in PyTorch 2.0, built on TorchDynamo (bytecode capture) and TorchInductor (code generation backend).

The torch.compile feature introduced in PyTorch 2.0 improves inference and training speed through graph capture and compilation optimization while maintaining the ease of use of dynamic graphs.

torch.compile delivers an average training speedup of 30% to 2x on mainstream models and can be enabled with just a single decorator call.

On GPUs that support Tensor Cores, combining torch.compile and SDPA typically delivers a 1.5–2x speedup without changing mathematical equivalence.

torch.compile optimizes a model's forward and backward passes through graph capture and kernel fusion.

PyTorch 2.x introduced torch.compile, which completes graph capture and operator fusion with almost no changes to user code.

TorchTPU completes graph compilation optimization through torch.compile and XLA, allowing developers to switch to compiled mode when performance is needed.

Inductor is the default compilation backend of torch.compile in PyTorch 2.x, responsible for splitting FX graphs into kernels, generating Triton or C++ code, and providing a pluggable backend interface for different accelerators.

On a single H100 with torch.compile, BF16 precision and two 256x256 RGB inputs, REKI produces a complete action chunk in only 21.8 milliseconds, 35.5% faster than π0.5 (33.8 ms).

TorchNeuron, as a native backend, is registered as an optional backend for torch.compile, enabling the same model code to be lowered to Trainium for execution without modification.

The installer's AI application matrix (ComfyUI, SwarmUI, Musubi Trainer, Ultimate TTS, etc.) depends on components that are sensitive to the compilation environment, such as torch.compile, Triton, flash-attention, sage-attention, xformers and insightface.

The Windows installer uses a scheduled task to re-point to the new Visual Studio compiler version at startup, at login, after VS updates and daily, so that the paths in PATH remain valid and features that depend on compilation, such as torch.compile, keep working.

The IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.

torch.compile is a just-in-time compilation (JIT) mechanism introduced in PyTorch 2.0, and its workflow has three steps: TorchDynamo captures the computation graph at the bytecode level, AOTAutograd handles the forward and backward graph structures, and finally it is handed to the backend compiler to generate executable code.

TorchInductor is PyTorch's default backend compiler, capable of lowering computation graphs into Triton GPU kernels or C++/OpenMP CPU code.

The torch.compile backend of PyTorch 2.0 extensively uses kernels generated by Triton.

TorchScript and torch.compile are used for model compilation and acceleration

In fullgraph mode, operations that rely on Python dynamic control flow or call external C++ extensions without FakeTensor registration will cause a graph break, causing compilation to fail.