Agentic Coding with Small Models: 100x Lower Cost, Double the Success Rate

In the field of AI coding, frontier large models (such as the Gemini series) are impressive, but for many everyday development tasks they can be too expensive—like using a sledgehammer to crack a nut. A talk from the Google team explored in depth how to build efficient agentic coding workflows using small language models (SLMs), striking a balance between cost, privacy, and latency.

Small Models Are Not Replacements, but Complementary Tools

The speaker Jose opened with a vivid analogy: frontier models are rockets, small models are bicycles. Rockets are fast, but bicycles are often more efficient for short commutes. They emphasized that small models are not meant to replace large models but to complement them—especially when models are open source and weights are downloadable, the advantages become even more obvious.

The cost difference is stark. On a dataset of 200 medium-to-high difficulty tasks consuming nearly 131 million tokens, using Gemma 4 costs less than $9 (called through a commercial gateway; free if run locally), while frontier models can cost 20 to 200 times more, depending on factors like token caching.

But cost is not the only consideration. On the same dataset, Gemma 4 can independently solve about 53% of tasks. This means that if you use a frontier model for all tasks, you may be wasting resources; if you only use small models, you'll need guidance or to switch to a larger model. The key is finding the right scenarios.

Know Your Toolbox: Small Models Come in Many Varieties

Before deciding which model to use, you need to know what options exist. The speaker Zoe pointed out that when most people think of language models, they first think of autoregressive models, which predict the next most likely token one at a time based on input tokens. But models are far more diverse than that.

Diffusion models (like Diffusion Gemma) process blocks of 256 tokens at a time, similar to sketching an outline first and then gradually focusing, which can reduce latency. Embedding models convert text or images into vectors, comparing conceptual similarity by computing distances—for example, determining whether a hot dog is a sandwich requires only simple math.

There are also models specialized for single tasks: function-calling models convert natural language into API calls, vision models process images, and classifiers score inputs. These models vary in size, context window, and memory footprint, and selection requires weighing trade-offs based on task requirements.

Autoregressive models are the foundation architecture of the vast majority of current language models (including GPT, Gemini, Llama series). Their core mechanism is "token-by-token prediction": given the preceding text, the model outputs a probability distribution for the next word, samples from it, and feeds it back as input to continue predicting until an end token is generated. This serial generation naturally introduces latency, because each token depends on the computation of the previous token. In contrast, diffusion models process a block of tokens (e.g., 256) in parallel at once, gradually refining the output through multiple rounds of "denoising," offering potential advantages in low-latency scenarios (such as code completion), but they are currently less mature than autoregressive models in long-text coherence. Understanding this distinction helps determine when to use which model.

Four Advantages of Running Small Models Locally

Why choose local models? The speaker Ian summarized four points:

• Privacy: Sensitive data such as API keys and proprietary code never leaves the local machine.
• Low latency: Once the model is loaded into memory, responses require no network round-trip.
• Zero marginal cost: Retries and error loops run locally incur no additional cost.
• Offline availability: Works in environments without network access, such as on planes or trains.

However, local operation faces two major challenges: memory and speed. Large models (e.g., 26B, 31B parameters) require over 50GB of VRAM at BF16 precision, making them difficult to run on ordinary hardware. Quantization techniques can reduce precision (e.g., to 4-bit), cutting memory requirements by 75% and making them fit on devices like Macs with unified memory.

Quantization-aware training (QAT) can compress models while reducing accuracy loss. For mobile devices, mixed quantization methods can shrink models further—for example, from 7.4GB for Gemma E2B to 1.1GB, enabling them to run on mid-to-high-end phones.

On the speed side, speculative decoding uses a small model to predict multiple tokens, which are then verified by a large model, sharing the KV cache. This can achieve about 3x speedup on predictable tasks like coding without quality loss.

Quantization refers to the process of converting model weights and activations from high-precision floating-point numbers (e.g., BF16/FP16, 2 bytes per parameter) to lower-precision representations (e.g., INT8, INT4, 1 or 0.5 bytes per parameter). Common post-training quantization (PTQ) directly truncates precision on an already-trained model—simple to implement but potentially lossy; quantization-aware training (QAT) simulates quantization errors during training so the model learns to compensate, preserving more capability at the same compression rate. 4-bit quantization can reduce memory usage to about one-quarter of BF16, allowing models that originally required over 50GB of VRAM to run on a Mac with 32GB unified memory. But quantization is not lossless: the lower the bit width and the more complex the task, the more noticeable the accuracy drop, so it's necessary to evaluate actual performance after quantization for specific tasks.

Speculative decoding is an inference optimization technique that accelerates autoregressive generation. The idea: use a smaller, faster "draft model" to predict several candidate tokens in succession, then have the target large model verify these tokens in parallel in a single forward pass—accepting matching parts, rejecting inconsistent parts, and resampling from the rejection point. Since the computational cost of the large model verifying multiple tokens is similar to generating a single token, overall throughput can be significantly improved. The key is that the draft model and target model share the vocabulary and KV cache, and the higher the acceptance rate of draft predictions, the greater the speedup. Coding tasks, due to highly predictable syntax and patterns, typically have high acceptance rates, thus achieving about 3x speedup with output quality identical to the original model (since the large model has the final say).

How to Evaluate Small Models: Beyond Benchmark Scores

Ian emphasized that looking at benchmark scores alone (such as Live Code Bench) only provides high-level metrics; what really needs evaluation is the entire stack: the model (weights and quantization method), the inference runtime (e.g., Ollama, LightRT), the agent framework (e.g., pi, OpenCode), and the context window.

A real case: in Android Bench testing, when using Ollama's default 32K context, the agent framework's compression mechanism moved files out of memory, causing task failure. After increasing the context to 256K, the model could keep files in memory, and the task's success rate doubled.

Therefore, they proposed the "Are We There Yet?" framework, which evaluates task readiness from four dimensions: model, inference engine, framework, and hardware configuration, considering factors like reliability, latency, and memory usage, ultimately giving green, yellow, or red ratings.

Context window refers to the maximum number of tokens (input plus output) a model can process in a single inference. When the accumulated conversation history, file contents, and tool outputs in an agent workflow exceed this window, the framework must adopt compression strategies: truncating old messages, summarizing history, or moving files out of context. These mechanisms can easily cause the model to "forget" key files or instructions in long tasks, leading to task failure. Increasing context from 32K to 256K essentially reduces the frequency of triggering compression, allowing the model to work with complete information. But a larger context also means higher memory usage and slower inference speed, so configuration should be balanced based on task type and hardware conditions rather than simply maximizing size.

In Practice: Small Models in Android Development

Jose classified tasks by difficulty into four categories: small tasks, medium tasks, large tasks, and long-horizon tasks. Small models perform excellently on small tasks, such as basic Compose optimization (one-line code change, 3 minutes). Medium tasks, like fixing request errors and retry logic in Pocket Casts, take 10 minutes, modify 3 files, and cost only $0.06 (via OpenRouter).

But small models have limitations. In a failed medium task, the model implemented almost all logic correctly but assumed an initial sync value of -1 (actually 0) and did not run existing unit tests to verify. This reflects small models' tendency to "stop early" rather than proactively pursuing perfection like frontier models.

About half of large tasks pass, but some failure cases show the model "completing 90%" while missing key requirements (e.g., not adding test cases for malformed URL parsing). Small models follow instructions but don't "try to impress you."

For long-horizon tasks (e.g., migrating an entire app, building a complex app from scratch), frontier models achieve only 3% to 40% success on Android Bench v2, and small models are clearly out of their depth. But routing can assign preliminary analysis to frontier models and cascade simple tasks to small models, achieving cost efficiency.

Practical Tips for Optimizing Small Model Workflows

To improve small model performance, there are several simple and effective adjustments:

• Configure the context window correctly: For example, Ollama defaults to 32K, but Gemma 4 supports 8x that. In evaluations, enabling the full context improved results from 26% to 53%.
• Use the Android CLI: For new or niche APIs, searching documentation directly is more effective than web search.
• Add verification prompts: In agents.md, tell the model how to run tests, manage emulators, etc., but avoid overly long files that degrade performance.
• Add or remove examples: Small models may need more time, but for new APIs, providing examples helps understanding; for general foundations, avoid stuffing too many examples.

Orchestration: Making Bicycles and Rockets Work Together

Zoe introduced various uses of orchestration: avoiding service interruptions or rate limits through fallback models; having small models consult "advisor" models when needed; having multiple models form a "committee" to execute tasks simultaneously and evaluate results; using another model for peer review; and achieving cost efficiency through cascading—for example, using a frontier model for planning, a small model for execution, and another model for testing.

Orchestration can be set up manually or automatically routed using lightweight LLM agents or classifiers. Ultimately, the quality gap between small and frontier models is not fixed but can be bridged through engineering.

Conclusion

The core of small model agentic coding workflows is: choose the right tools, adjust task scope, test the entire system, and orchestrate flexibly. Through open-source small models like Gemma 4, developers can gain privacy, low latency, and zero marginal cost locally, while leveraging orchestration to complement frontier models for high-performance AI-assisted development. As the speakers said, you can have both bicycles and rockets—the key is how to use them together.

One-line Summary

Google team shares how to build low-cost, locally runnable agentic coding workflows with small language models, complementing frontier models through orchestration.

Paragraph Summary

Based on a Google team talk, the article explores building agentic coding workflows with small language models (SLMs). Small models can be 20-200x cheaper than frontier models, with Gemma 4 independently solving about 53% of tasks on the test set. Local operation brings four advantages: privacy, low latency, zero marginal cost, and offline availability, while quantization can reduce memory requirements by 75%. Evaluation must cover the entire stack: model, inference engine, framework, and hardware—improper context configuration can halve success rates. Through techniques like optimizing context windows and adding verification prompts, success rates can rise from 26% to 53%. Orchestration strategies (cascading, advisor models, committees) can combine the strengths of both model types to balance cost and quality.