Mobile AI Factory: Using AI Agents to Build Custom Small Models for Phones

When Small Models Meet Phones: Customization Is the Only Way Forward

Putting language models on phones has inherent advantages in privacy and offline availability. But small language models (SLMs) face a practical problem: they only perform well when fine-tuned for a specific task and adapted to the hardware specs of the target device. And completing this tuning still relies on experts' repeated trial and error.

A new study proposes Mobile AI Factory—an agent system that takes a task description and a target phone as input and automatically produces a fine-tuned and quantized specialized small model.

Orchestrator + Fixed Rules Collaboration Mechanism

The core of the system is an orchestrator LLM: based on current progress, it decides whether the next step is tuning hyperparameters, mining real data, or generating synthetic samples. And once a change causes accuracy to drop, fixed rules handle the rollback. This "LLM decision + rule fallback" design separates the flexibility of exploration from stability.

The orchestrator LLM is the decision-making hub of Mobile AI Factory. Its role is not to directly train models, but to choose among actions such as "tuning hyperparameters," "mining real data," and "generating synthetic samples" based on the search history and current metrics. It divides labor with fixed rollback rules: the LLM proposes changes that may improve accuracy, and the rule layer automatically reverts changes when they cause metrics to decline. This design draws on the common "propose—verify" paradigm in agent systems, decoupling open-ended exploration from deterministic protection and preventing the LLM's randomness from directly destroying an existing optimal configuration.

Search Strategy: Start Small, Adjust Targets Dynamically

The factory searches across different base models and quantization schemes, trying model sizes from small to large progressively. During the search, the accuracy target itself also adapts—there's no need to lock in a possibly unrealistic metric from the start.

An easily overlooked detail: the system calls real measurements on the target phone, covering different inference engines and backends. In other words, the basis for selection is not paper specs but actual measured performance.

Measured Results Are Convincing

The study evaluated 10 tasks on 4 Android phones. On unseen test sets, its on-device models outperformed two five-shot frontier models on 5 tasks, leading by up to 0.34, and tied on 3 others.

Search efficiency is equally noteworthy: on an entity extraction task, the gap between the automated search result and exhaustive training of all 27 model variants was only 0.03, while saving 19x GPU time.

Quantization refers to converting model weights and activations from high-precision floating point (e.g., FP16) to low-precision integers (e.g., INT8, INT4), with the goal of compressing model size, reducing memory usage, and leveraging the integer acceleration capabilities of phone NPUs. Different quantization schemes (such as post-training quantization vs. quantization-aware training) have significantly different impacts on accuracy, and the same scheme may also perform differently on different inference engines. Therefore, Mobile AI Factory incorporates quantization schemes into the search space and decides them jointly with base model size, rather than selecting a quantization configuration separately after fine-tuning is complete.

Why Phone Measurements Are Irreplaceable

The study offers a blunt conclusion: the best configuration changes with the phone. Without real-device measurements, even giving the full hardware and model specs to Claude Opus 5.5 still results in it missing application requirements in 25% of simple and medium cases, rising to 60% in difficult cases.

This explains why relying solely on model reasoning for selection is unreliable—actual performance differences across hardware and backends must be captured through measurement. The code is open-sourced on GitHub.

Note: The comparison model mentioned in the original text is Claude Opus 5.5, cited here as-is from the source material.

Five-shot frontier models refer to large-scale frontier models that are provided with only 5 examples for in-context learning during evaluation. The significance of this comparison lies in measuring whether on-device specialized small models can approach the task performance of general large models under very few supervision signals. Since large models cannot be deployed offline on phones and incur high inference costs per call, if on-device small models can match or exceed five-shot large models on some tasks, it shows that task- and hardware-specific tuning can compensate for the gap in parameter scale.

One-line Summary

Mobile AI Factory uses an orchestrator LLM plus fixed rollback rules to automatically produce fine-tuned, quantized specialized small models for a given phone.

Paragraph Summary

Mobile AI Factory is an agent system that takes a task description and target phone as input and automatically performs hyperparameter tuning, real data mining, and synthetic data generation to produce a fine-tuned and quantized specialized small model. Its core is an orchestrator LLM that makes decisions and fixed rules that roll back when accuracy drops. The system searches across base models and quantization schemes from small to large, dynamically adjusts accuracy targets, and calls real-device measurements covering different inference engines and backends. Across 10 tasks and 4 Android phones, on-device models outperformed two five-shot frontier models on 5 tasks, leading by up to 0.34 and tying on 3; on an entity extraction task, the gap versus exhaustive search over 27 variants was only 0.03 while saving 19x GPU time. The study also emphasizes that real-device measurement is irreplaceable and that the best configuration changes with the phone.