On-Device Inference

Current mainstream mobile devices typically have 6–16GB of memory (with approximately 3–8GB available for models), and models that can run smoothly on-device typically range from 1B to 8B parameters, requiring approximately 2–5GB of memory after 4-bit quantization.

Locally deployed models download AI model weights to the user's hardware for execution, with inference completed locally so that data does not need to pass through external servers.

The cost of local inference is that model size is limited by the user's hardware, and performance depends on the device.

Inference runs on the user's device, so websites do not need to pay cloud inference costs for each call, nor are they subject to the quotas and rate limits of third-party APIs.

On-device inference typically can only run smaller models (on the order of hundreds of millions to billions of parameters), and first-token latency and throughput are strongly device-dependent.

Compared with cloud-based large models, on-device inference eliminates upload latency, making "zero data egress" possible, but it requires rebalancing among model size, inference speed, and recognition accuracy.

The core difference between on-device inference and cloud inference is that the constraints change: on-device inference is limited by power budgets, thermal design, memory bandwidth, and heat dissipation conditions, models usually need to be quantized and sparsified, and inference latency must be stable and predictable.

The heat and power consumption performance of multimodal inference on phones directly affects whether it can stay resident in an application.

The most common problem in on-device inference is memory, as context length silently inflates the KV cache until it collides with the weights.

503s returned during the loading process should be treated as normal signals rather than failures, and backoff-retry should be applied to the first request.

In on-device LLM inference, model weights, KV cache, and runtime must all fit simultaneously within the device's fixed memory.

On-device inference context length should be determined by the device's available memory, not by the model's theoretical maximum.

On-device inference has a cold-start latency where the model must first be loaded into memory, and this time may dominate the user's first impression.

On-device devices do not auto-scale, so real memory headroom should be measured before release.

Mean rather than median should be used to estimate on-device throughput, because the median filters out slow but real use cases, making capacity estimates overly optimistic.

On-device devices may have multiple bottlenecks simultaneously—CPU, GPU/NPU, memory bandwidth, etc.—so one should observe which resource saturates under load rather than blindly optimizing inference speed.

On-device inference is worth adopting when: data cannot leave the device, the network is unreliable, or latency cannot depend on network round-trips.

On-device inference runs the model directly on the user's local device, with the advantages of extremely low latency, data remaining on-device, and offline availability.

The benefits of local deployment include zero-latency experience, zero-cost operation, controllable privacy, and offline availability.

For tasks requiring real-time decision-making, the low-latency advantage of local deployment may outweigh the computational power advantage of the cloud.