Android ML Kit On-Device AI: Performance Optimization and Quality Breakthroughs

Why Choose On-Device AI: Four Core Advantages

With cloud AI capabilities advancing rapidly, why run AI inference on the device itself? David Cho, Product Lead for Google Android AI Core, gave four key reasons in a technical talk: speed, cost, availability, and privacy.

On speed, on-device inference avoids transferring data back and forth between the user's device and the cloud. According to his testing, on-device AI runs up to 2x faster than cloud in many scenarios. On cost, assuming a million inferences per day, running on user devices can save hundreds of thousands of dollars compared to the cloud. On availability, a developer survey showed that 62% of developers listed offline availability as one of their top three reasons for using on-device AI—whether a user is at 10,000 meters in the air or in the underground levels of the Tokyo subway, AI capabilities remain accessible. Privacy is another key driver: 72% of developers listed it among their top three reasons, and 40% listed it as their number one reason. Users prefer that their text messages, voice notes, photos, and even financial and medical information stay processed locally.

ML Kit: Making On-Device AI Accessible

According to David Cho, ML Kit is designed to greatly simplify the integration of on-device AI. Developers only need to call the ML Kit API; the underlying complexity—managing input/output safety signals, downloading models and weights, runtime configuration—is all handled by AI Core.

AI Core currently covers 175 million devices and is expected to expand significantly to hundreds of millions of new devices over the next year. On the ecosystem side, 9 third-party apps with over 10 million downloads each have already adopted these generative AI capabilities, and app launches of AI production features grew 110% quarter over quarter.

Four Underlying Optimizations of Gemini Nano

To make the on-device model more efficient, Google made several deep optimizations to Gemini Nano that significantly reduce runtime memory requirements. David Cho listed four core improvements:

• 2-bit quantization replacing 4-bit quantization: halving the bit count means halving the amount of data that needs to flow through the model.
• Shared KV cache: the first 20 layers of the model share the KV cache, greatly reducing the overall package size while still supporting a 16K token context length.
• Lazy loading of vision and audio encoders: for the many text-only use cases, these encoders don't need to be loaded upfront, saving memory.
• NPU clock scaling: letting the NPU run on demand to save power while ensuring high-quality output.

On speed optimization, prefix caching means system prompts don't need to be reprocessed each time, saving 60% of input tokens in the example, corresponding to a 60% speedup. Speculative decoding uses a draft model far smaller than Gemini Nano to rapidly generate multiple tokens, which Gemini Nano then verifies, achieving an average 2x speedup in internal use cases.

Quantization is a core technique for compressing neural network models: it maps model weights and activations from high-precision floating point (e.g., 16-bit or 32-bit) to lower-precision integer representations, thereby reducing storage footprint and memory bandwidth requirements. Common approaches include post-training quantization (PTQ) and quantization-aware training (QAT). The lower the bit count, the smaller and faster the model, but the greater the risk of accuracy loss. 2-bit quantization is extremely low-bit quantization, typically requiring special scaling factors, grouping strategies, or mixed-precision handling to maintain usable quality; compared to 4-bit quantization, its theoretical memory footprint and transfer volume are halved—this is the technical background behind the "halving the data" statement in the text.

KV cache (Key-Value Cache) is a key optimization in Transformer autoregressive inference: when generating each new token, the model needs to recompute the attention key-value pairs for all previous tokens; caching these results avoids redundant computation and significantly accelerates long-context generation. However, the KV cache grows linearly with context length and consumes substantial memory. Shared KV cache (i.e., multiple layers reusing the same key-value representation) is an architectural trade-off that sacrifices a small amount of quality for substantial memory savings, enabling small models to support long 16K-token contexts.

Prefix Caching targets the system prompts that recur repeatedly during inference. In chat or agent applications, system instructions, role settings, and examples are often identical across requests; re-encoding these prefix tokens each time would cause significant redundant computation. Prefix caching saves the KV state of already-processed prefixes and reuses it directly in subsequent requests, skipping the redundant prefill phase. The "60% of input tokens saved" in the text refers to these prefix tokens no longer needing recomputation, corresponding to roughly a 60% latency reduction.

Speculative Decoding is an inference technique that accelerates autoregressive generation: a draft model with far fewer parameters than the main model (here, Gemini Nano) rapidly generates multiple candidate tokens in succession, and the main model then verifies in parallel whether these tokens are acceptable. If the draft model's predictions match the main model, multiple tokens are confirmed at once, breaking the bottleneck of serial token-by-token generation; if not, it falls back to the main model to regenerate. This method preserves the same output distribution as the main model (lossless); the actual speedup depends on the draft model's acceptance rate, and the internal use cases in the text achieve an average 2x.

Quality Leap: 2B Model Rivals Last Year's 27B

One impressive development is the improvement in model quality. David Cho announced that this 2B-parameter model can now compete with last year's 27B model on multiple benchmarks. On the standard knowledge benchmark MMLU Pro, the 2B model nearly matches last year's 27B model; on graduate-level scientific reasoning and agentic tool-use benchmarks, the 2B model even performs better, despite being less than one-tenth the size of last year's model.

David Cho said this trend is expected to continue, and the team is working closely with the GDM team to keep improving the model.

Capability Expansion: From Speech Translation to Function Calling

ML Kit recently added four key capabilities: speech translation (supporting transcription in over 20 languages), embeddings (enabling search of app content by semantics rather than keywords), text-to-speech (providing near-human voice), and function calling (letting apps perform actions on behalf of users). In addition, capabilities such as system instructions, multi-image input, thinking mode, and structured output are also ready. Structured output allows developers to annotate Kotlin classes and directly return objects conforming to a schema.

Embedding is a representation method that maps text, images, or audio into high-dimensional dense vectors, where semantically similar content is closer together in vector space. Multimodal embeddings map different modalities into the same semantic space, so that "lens order email" and "lens product image" can be compared even without shared vocabulary. Retrieval typically uses cosine similarity to measure the angle between vectors; the threshold of 0.68 in the text is the boundary for determining "similar enough." Embeddings are the infrastructure for semantic search, recommendation, clustering, and retrieval-augmented generation (RAG).

Function Calling lets a large model, rather than executing code directly, output a structured invocation intent (function name and parameters), with the host application deciding whether to execute it and return the result. This division of labor—"model proposes, app executes"—is key to on-device privacy and security: the model is only responsible for understanding and decision-making, while sensitive data operations remain in app-controlled code. Combined with Thinking Mode—which lets the model generate intermediate reasoning steps before giving a final answer—this can significantly improve the accuracy of complex tool selection and parameter filling; the 13-percentage-point improvement in the text stems from this.

End-to-End in Action: Cross-Border Package Scam Detection

David Cho used a complete case to demonstrate how to chain all the APIs together. The scenario: a U.S. photographer traveling in Japan receives a voice mail and text message claiming a 6,800 yen customs clearance fee is due, and wants to determine whether it's a scam, while requiring that the information be private, explainable, and accessible.

The entire flow is divided into four stages:

Information Ingestion: Use the speech translation API to transcribe and translate the Japanese voice mail, while using the speech recognition API to process the user's question. Both share the same audio encoder trained on over 100 languages.

Triage: Through the Prompt API and function calling, combined with system instructions (e.g., a "security auditor" role) and thinking mode, determine what information needs to be retrieved. David Cho specifically emphasized that using thinking mode during function calling yields a 13-percentage-point accuracy improvement on a 100-point scale.

Retrieval: Use multimodal embeddings and App Search. When the user ordered the lens, the order email and image were indexed as embedding vectors; when verification is needed, cosine similarity is used for retrieval with a threshold of 0.68. The value of multimodal embeddings is that even when images and text share no vocabulary, they can be mapped into the same semantic space. As a clarification, Gemini Nano does not execute any code—it only formulates queries, and the actual retrieval is performed by app code, ensuring privacy and security.

Audit and Feedback: Generate embeddings again, compare the retrieved order information with the incoming call information, and use the Prompt API, structured output, and speech synthesis to give a verdict. The output class defines visual contradiction types (e.g., tariff fraud, packaging mismatch, status conflict) and explanation fields. The final output includes a scam verdict, fraud risk score (0-100), amount saved, and a list of contradictions. Finally, text-to-speech reads the result aloud, generating a response in under 120 milliseconds on most devices, with platform APIs as a fallback.

The Scalable Future of On-Device AI

This case fully demonstrates the combined advantages of on-device AI in speed, cost, availability, and privacy. David Cho said these capabilities have rolled out over roughly the past six months, and developers can find detailed information in the official documentation. As device coverage expands and model efficiency continues to improve, on-device AI is moving from proof of concept to large-scale production deployment.

One-line Summary

Google details the four advantages of on-device AI, Gemini Nano optimizations, new ML Kit capabilities, and a real-world scam detection case.

Paragraph Summary

Based on a technical talk by David Cho, Product Lead for Google Android AI Core, the article systematically explains the four advantages of on-device AI: roughly 2x faster than cloud, hundreds of thousands of dollars saved per million inferences, offline availability (valued by 62% of developers), and privacy protection (valued by 72% of developers). ML Kit lowers the integration barrier by encapsulating complexities such as model downloads and runtime configuration, and AI Core already covers 175 million devices. Gemini Nano reduces memory through 2-bit quantization, shared KV cache, lazy loading of encoders, and NPU clock scaling, while speeding up via prefix caching and speculative decoding. The 2B model rivals last year's 27B model on MMLU Pro. New capabilities include speech translation, embeddings, text-to-speech, and function calling, demonstrated through an end-to-end cross-border package scam detection case, all completed on-device.