Formula E AI Co-Driver: Gemini Nano on Your Phone in the Cockpit

Title

Formula E AI Co-Driver: Gemini Nano on Your Phone in the Cockpit

Meta Description

Google Cloud deployed an on-device AI co-driver in a Formula E car: a Raspberry Pi reads CAN bus telemetry, a Pixel phone runs Gemini Nano via ML Kit, with ADK and A2A enabling real-time voice Q&A—including hard-won lessons on heat and Bluetooth at Goodwood.

Article Content

Packing a large model into the cockpit of a race car doing 200 km/h and having it answer driver questions in real time without a stable network connection—that sounds like science fiction, but the Google Cloud team actually pulled it off at the Goodwood Festival of Speed. In a recent tech talk, Alistair from the Google Cloud CTO Office and Oli from Formula E broke down the project's tech stack and the pitfalls they hit along the way.

Why Run an On-Device Agent in a Race Car?

The starting point was clear: validate whether Agents are viable in extreme edge environments. The cockpit is hot, Bluetooth signals are blocked by the carbon fiber body (in the speakers' words, "this car is basically a Faraday cage"), hundreds of thousands of spectators at the venue cause network congestion, and a driver traveling over 200 km/h has very little tolerance for response latency—it's almost a checklist of every nightmare scenario for on-device AI deployment.

But the team quickly realized that all the "ingredients" they needed already existed: Android as the host platform, ADK (Agent Developer Kit) for Agent logic, Gemini Nano called through ML Kit for the on-device model, plus fully offline speech recognition and TTS APIs. The TPU built into the Google Pixel phone provides hardware acceleration for the on-device model, so the solution didn't require any custom microcontrollers.

System Architecture: From CAN Bus to Cockpit Conversation

Data Pipeline: Raspberry Pi as the "Translator"

The vehicle's thousands of sensor data points are aggregated through the CAN bus to the VCU (Vehicle Control Unit). The team connected a Raspberry Pi directly to the VCU, where it decodes the raw signals, applies transformations, and downsamples to about 10 Hz—more than enough for the co-driver use case. The Pi then publishes the refined data via the MQTT protocol to a Pixel 10 Pro in the cockpit.

CAN bus (Controller Area Network) is a serial communication bus standard widely used in automotive electronics. It uses differential signal transmission, offers strong resistance to electromagnetic interference, and connects the vehicle's ECUs, sensors, and actuators. Raw CAN frames contain only an ID and up to 8 bytes of payload; the specific meaning is defined by each manufacturer's DBC file, so dedicated decoding logic is needed to map bits to physical quantities such as vehicle speed and throttle position. The VCU (Vehicle Control Unit) is the core control node in an electric vehicle, responsible for coordinating the powertrain and aggregating overall vehicle status. Because CAN data is high-frequency and high-dimensional, sending it directly to the phone would place a burden on the device—which is why the project performs transformation and downsampling on the Pi.

Agent Runtime: ADK + Gemini Nano

The phone app builds the Agent on ADK for Android, with Gemini Nano hosted by ML Kit as the core model. There are two input paths: telemetry data coming in via MQTT, and the driver's speech transcribed by Android speech recognition. The Agent's output also goes in two directions: talking to the driver directly via TTS, or calling a cloud Agent through the A2A (Agent-to-Agent) protocol for more complex tasks—such as querying the current race position, information the race car itself doesn't have and that requires cloud collaboration.

For interaction design, the driver says a wake word and asks a question; after answering, the Agent keeps listening for about five seconds to support follow-ups, mimicking the conversational rhythm of a real race engineer. Multi-turn dialogue lets the Agent access historical context and keep the conversation coherent.

A2A (Agent-to-Agent) is an open protocol led by Google, designed to let AI Agents built on different frameworks and by different vendors discover, communicate with, and collaborate with one another. It defines a standardized message format for exchanging tasks, states, and results between Agents, typically implemented over HTTP/JSON. In this project, the in-car on-device Agent uses A2A to delegate information the race car doesn't have (such as real-time race standings) to a cloud Agent, which processes it and sends the result back—forming a device-cloud division of labor. This is distinct from MCP (Model Context Protocol, which connects models to tools/data sources): MCP addresses how an Agent calls tools, while A2A addresses how Agents call each other.

Gemini Nano is the lightweight, on-device-oriented member of the Google Gemini model family, offered in tiers such as Nano-1 and Nano-2 by parameter count, quantized and distilled to fit within a phone's memory and compute budget. It is exposed to Android apps through ML Kit (Google's mobile machine learning SDK) as a GenAI API, and inference can be accelerated by the TPU built into Pixel devices, all without data leaving the device. Compared with cloud large models, Nano has a limited context window and breadth of knowledge, making it suitable for tasks like intent understanding, summarization, and short dialogue, while complex queries need to fall back to the cloud—which is exactly why this project adopted a hybrid device-cloud architecture.

Code Structure: Simpler Than You'd Expect

The speakers emphasized that the code isn't complex: a few lines to call ML Kit and load Gemini Nano, and ADK to define the Agent name and two types of instructions. Static instructions define the Agent's persona and track knowledge, while dynamic instructions are generated on each query based on the latest telemetry summary and conversation history, ensuring the Agent always knows the driver's current state. At the tool level, three capabilities are registered: session data recall, cloud Agent invocation, and TTS voice output.

Real-Vehicle Validation: Data Flowed, but the Trouble Was Just Beginning

The demo on the simulation rig was smooth—the driver asked "how aggressive was my steering," and the Agent answered a peak steering angle of 48 degrees and an average of 11 degrees per second; asked about peak G-force, it answered 2.2G; asked about overall performance, it answered a top speed of 306 km/h, an average of 186 km/h over 35 seconds, and full throttle 58% of the time.

But deploying in a real car was another matter. The team went to Spark Racing Technologies in Paris and connected the Raspberry Pi to the real VCU. All previous testing had been based on replayed data; only after powering up and waiting for telemetry to actually flow did they confirm the system's pipeline was connected.

At Goodwood, the first test was thermal management. The day happened to be one of the hottest of the year, but unexpectedly the phone itself ran fine—it was the Raspberry Pi that couldn't take the heat, and in the end they got through the hill climb by repeatedly recharging from an external UPS battery pack all day. The second test was the Bluetooth helmet connection, which had never been tested in the real world. The team sat on the grass debugging frantically with a laptop—relaxed on the surface, but in reality "one of the most stressful periods of my career"—and finally fixed the issue before the run using a hotfix and Android debugging tools.

Three Engineering Lessons

Android is a suitable host for on-device Agents. The combination of ADK, on-device models, offline speech APIs, and debugging tools lets a single phone deliver a complete Agent experience with no custom hardware.

Integration hell still exists. Getting each subsystem working on its own doesn't mean they'll work when put together. The driver's helmet Bluetooth connection is a typical case—the problem came from the least conspicuous link, and the importance of testing cannot be overstated.

Gemini Nano needs polishing, not plug-and-play. The speakers called it the project's "hero," but stressed that prompt engineering is essential: how to divide work between static and dynamic instructions, how to define and invoke tools, and how to run evaluation tests—all of these determine whether an on-device model can produce stable output in demanding scenarios.

The value of this project isn't the race car itself, but what it proves: when the network is unreliable, latency requirements are stringent, and the environment is harsh, an on-device Agent on a phone is already practically usable—and the toolchain you need is available today.

One-line Summary

Google Cloud and Formula E ran an on-device Agent in a race car cockpit at Goodwood for offline Q&A.

Paragraph Summary

The article details how the Google Cloud team, working with Formula E, put an on-device AI Agent into a race car cockpit at the Goodwood Festival of Speed. The system uses a Raspberry Pi to decode CAN bus telemetry and downsample it to 10 Hz, sending it over MQTT to a Pixel phone in the cockpit; on the phone, an Agent built with ADK calls Gemini Nano via ML Kit for inference, supporting offline speech recognition, TTS output, and A2A protocol calls to a cloud Agent for external information such as race standings, and is designed to keep listening for about five seconds after answering to support follow-ups. Real-vehicle deployment exposed integration issues such as thermal management and Bluetooth helmet connectivity, which the team solved with UPS recharging and on-site hotfixes. The article closes with three lessons: Android is well suited as an on-device Agent host, integration hell is real, and Gemini Nano requires targeted prompt engineering.