Core ML

Apple Intelligence是苹果在WWDC 2024上发布的系统级AI框架

The Apple Core ML framework allows developers to convert models trained in PyTorch or TensorFlow to the .mlmodel format for local execution, leveraging parallel computation across the GPU and Neural Engine.

The Neural Engine in Apple's custom chips is an NPU optimized specifically for matrix multiplication and convolution operations, and offers higher energy efficiency than general-purpose CPUs and GPUs on specific AI inference tasks.

在Apple Silicon设备上，借助Core ML转换工具和ANE加速，Whisper medium模型可实现接近实时的转录速度

Apple has simultaneous control over chip design, the operating system, and application frameworks (such as Core ML), enabling full-stack optimization from model compression to hardware scheduling.

Apple introduced an enhanced Speech framework in macOS 14, along with optimized Core ML conversions for the Whisper model.

Apple provides Core ML, Metal and the MLX framework launched in recent years in macOS and iOS for local model deployment.

The Neural Engine is a dedicated module in Apple Silicon specifically for accelerating machine learning computations, and it can be invoked through the Core ML framework.

For the iOS ecosystem, the barrier to integrating AI is lowering, and developers can obtain AI capabilities by calling cloud model APIs or using the on-device machine learning frameworks provided by Apple, without having to train models themselves.

On-device frameworks (such as Apple's Core ML and Create ML) package models directly into the app and perform inference locally on the device; the advantages are low latency, offline availability, and data never leaving the device, while the drawbacks are that model size and compute power are limited by phone hardware, usually requiring quantization and compression beforehand.

Early Neural Engines could only be invoked indirectly through Core ML, with a limited set of expressible operators; only newer platforms gradually opened up lower-level access.

Apple's Core ML framework can accelerate inference of speech recognition models on Mac, but the performance gap between M-series and Intel chips affects the real-time performance and accuracy of transcription.

When choosing tools, the usual first question is whether the task requires a custom model: general tasks prioritize ready-made APIs like ML Kit, and when fine-tuning on private data or domain-specific accuracy is needed, then take the custom model path of Core ML or TensorFlow Lite.

Core ML is the on-device machine learning framework provided by Apple; developers can convert trained models into .mlmodel format and integrate them into iOS applications, and the system automatically schedules the CPU, GPU, and Neural Engine to accelerate inference.

Apple's Metal Performance Shaders and Core ML frameworks can offload inference to the GPU and neural engine, reducing CPU usage.

Running the Whisper model on iOS devices requires converting the model to Core ML or using inference frameworks such as whisper.cpp, and leveraging the Apple Neural Engine for acceleration.

Achieving 13-second transcription of one hour of audio on a MacBook Air usually involves technical approaches such as model selection (the Whisper series or optimized versions), inference acceleration (Metal acceleration, Core ML conversion, etc.) and precision trade-offs (quantization such as int8, int4).

Anvil's image generation feature is called Anvil Dream, an SDXL Lightning variant converted to Core ML format via Apple's ml-stable-diffusion, generating 1024x1024 images on the neural engine.

ml-stable-diffusion is Apple's officially open-sourced conversion tool library, used to convert PyTorch weights of the Stable Diffusion series into Core ML format, enabling models to invoke the iPhone's neural engine for inference.

The test audio processing path is the same as microphone capture: FLAC is first decoded via AVAudioFile, then resampled with AVAudioConverter to 16 kHz mono Float32, and the audio is read into memory in advance so file I/O is not counted in the timing; only the transcribe stage is timed, and model download and Core ML preparation are outside the timing region.