ExecuTorch

ExecuTorch is an on-device inference runtime framework launched by Meta, built on the Export mechanism of PyTorch 2.0.

In ExecuTorch, operator lowering typically goes through multiple intermediate representations (such as Edge Dialect), ultimately mapping to Arm CPU-specific kernels or delegate backends.

On-device optimization of streaming ASR has been included as a key direction for ExecuTorch, and Arm devices are one of its explicit target platforms.

For streaming ASR, some stateful or dynamic-shape operators may not have ready-made efficient implementations and require additional adaptation or custom lowering.

ExecuTorch adopts a layered runtime design, including the representation layer, the operator layer and the backend layer (Backend Delegate).

ExecuTorch introduces a unified intermediate representation, EXIR; models are exported to EXIR via torch.export, then packaged in the .pte format and handed to the runtime for execution.

ExecuTorch is the on-device inference runtime officially launched by PyTorch in 2023, covering phones, embedded devices and even MCUs.

ExecuTorch solves the "deployment path" problem, not the "insufficient compute" problem; model size, operator support scope and inference latency are still limited by the chip itself.

Through memory planning, ExecuTorch computes the lifetime and offset of each tensor before execution, letting intermediate results reuse the same static buffer and avoiding runtime malloc/free.

ExecuTorch supports Arm Cortex-M microcontrollers, allowing PyTorch models to run on MCUs without conversion through intermediate formats such as ONNX.

Google's MediaPipe LLM Inference, Meta's ExecuTorch, and llama.cpp with Android support are mature mobile inference frameworks capable of mapping model computations to a phone's GPU or NPU for execution.

Do not trust availability flags self-reported by engines; make one real single-token probe call and cache the result.

On a structured output task involving a store audit checklist, ExecuTorch (Llama 3.2 1B, schema stuffed into the prompt) produced only 2 valid outputs out of 20 calls, while Apple Foundation Models (guided generation) produced valid outputs for all 20 calls.

Apple Foundation Models constrains model output at the framework level via guided generation to necessarily conform to a given type structure, whereas ExecuTorch has no corresponding capability and can only write the schema into the prompt.

ExecuTorch 1B on a simulator requires a 1.14 GB download (6 minutes 12 seconds), approximately 5 seconds to load on each launch, approximately 907 MB of in-app memory during generation, a generation speed of 43 tokens/s, and a median time-to-first-token of 579 ms for subsequent calls.

React Native 0.87 has removed RCTCxxBridge, requiring a patch to use RCTTurboModuleWithJSIBindings for injecting JSI bindings before ExecuTorch-related builds can compile.

ExecuTorch 0.10 reports anonymous download analytics by default, and setTelemetryEnabled(false) must be called to disable it.

The core design goal of ExecuTorch is to export trained PyTorch models into a format that can run efficiently on resource-constrained devices, covering scenarios such as smartphones, microcontrollers, and even wearable devices.

As a core supplier of edge computing chips, Arm has an incentive to participate in advancing the ExecuTorch learning ecosystem in order to lower the barrier for developers deploying AI models on its platform.

Every link in the ExecuTorch model deployment pipeline may encounter issues such as unsupported operators, precision loss, or insufficient performance.