On-Device AI

On-device processing means the AI model runs inference and computation directly on the user's device, without uploading raw data to the cloud.

Locally running models make trade-offs in inference depth, context length, and complex analytical capabilities compared to cloud-based models with hundreds of billions of parameters.

The advantages of on-device AI lie in low latency, privacy protection and offline availability, but limited by device compute and storage, its models are usually far smaller than cloud-based large models.

Siri's AI capabilities rely heavily on on-device processing (on-device AI).

Each additional language supported by Siri requires retraining, compressing and deploying the corresponding speech recognition and semantic understanding models under limited device resource constraints.

Local running does not automatically equal full compliance: the model's provenance, the logging approach, and the app's own network behavior still need to be assessed separately, so such features are typically seen as a means of reducing compliance friction rather than a sufficient condition for compliance.

Whether an agent can run locally depends not only on whether the model can perform local inference, but also on whether the tool-calling chain, permission controls, and execution environment are all closed-loop on the device side.

This move by Perplexity continues the trend of AI products migrating to the device side; running the agent locally addresses enterprise users' data compliance concerns and reduces reliance on network and cloud compute.

On-device running confines data processing to employee devices, reducing the risk of data being retained on third-party servers and making it easier to audit through enterprise endpoint management policies.

The performance ceiling of local running depends on the device hardware, and handling capability and response speed on complex tasks may differ from the cloud version; the official side has not yet provided comparison data.

In the past, optimization of on-device large models focused mostly on quantization and pruning, but system-level scheduling and prefetching can likewise unlock enormous headroom.

On-device agents place noticeably different demands on models than cloud agents, and low latency and stable structured output are often more critical than the absolute level of intelligence.

Two constraints of on-device AI are that the chip needs to retain compute redundancy, and that the model needs to be compressed for limited memory and power consumption.

The three main practical constraints facing on-device AI are: hard memory constraints (the two models cannot coexist), the hidden costs of heat dissipation and battery life (continuous inference on the neural engine generates heat and consumes high power), and the operational difficulty of model updates (users need to re-download hundreds of MB or even GB-level weights).

Scenarios suitable for on-device AI include: processing personal data that is extremely privacy-sensitive, environments where the network is unstable or unavailable, and simple interactions that require low-latency responses; unsuitable scenarios include: complex reasoning, large-scale knowledge retrieval, and applications that require frequent updates to model capabilities.

The article argues that on-device AI, thanks to the combination of small models, quantization, and mobile inference frameworks, enables amateur developers to run AI with interactive capabilities on consumer-grade Android devices without a server.

Running offline brings benefits such as privacy protection, no network dependency, and zero API costs.

无人机AI系统通常依赖边缘计算，在本机上运行轻量化计算机视觉模型而非依赖后方服务器