On-Device LLM + ESP32: Build a DIY Calorie Reminder

Stick a small screen on your fridge door, and every time you open the fridge you can see your calorie intake for the day at a glance — this idea sounds like some smart-home gadget, but at its core it's just an ordinary phone and an ESP32. Bilibili creator "电子博Studio" recently open-sourced a project that connects photo-based calorie estimation with a hardware reminder.

Phone side: recognizing food with a local model

The core of this project is the photo-based calorie estimation feature in the phone app. According to the creator, it uses the open-source miniCPM V4.7 35B model, whose key feature is that it can be deployed directly on the phone for on-device image understanding.

The workflow is straightforward: take a photo of the food you're eating, and the model identifies the types of food in the image and gives a calorie estimate. For example, snap a photo of your cafeteria lunch, and it completes the recognition and estimation.

Running the model locally means images don't need to be uploaded to the cloud, which is better for privacy and avoids network latency. For high-frequency, lightweight scenarios like "snap a quick photo," on-device inference makes the experience feel more seamless.

miniCPM is a series of open-source on-device large models launched by ModelBest in collaboration with Tsinghua University's NLP lab, focused on small parameter counts and high performance, capable of running offline on phones, tablets, and other edge devices. The "V4.7 35B" naming may correspond to a specific version and parameter scale, but note: 35B (35 billion parameters) is a heavy burden for on-device phone deployment, usually requiring quantization/compression or relying on high-performance chips; what actually runs on phones is typically the 1B–3B miniCPM variants. These models understand image content through a vision encoder and then output recognition results in natural language, which is how they can perform "describe-what-you-see" food recognition and calorie estimation. The core advantages of on-device inference are privacy protection and low latency, but the cost is storage space and compute, with clear thresholds for device memory and chip performance.

ESP32 mini screen: putting the data right in front of you

The second part of the project is a small ESP32 screen. It syncs data from the app and uses green, yellow, and red rings to show your current calorie intake status.

This design turns abstract numbers into color signals: green means normal, yellow warns you're approaching the limit, and red means you've exceeded it. Users don't need to open their phone to check specific values — a glance at the color is enough to make a judgment.

The creator also added a magnet to the back of the screen, so it can stick to the fridge door, or sit next to the coffee machine or somewhere in the snack area.

This placement choice is actually key. If a calorie reminder only lives on your phone, it's easy to ignore; but placing it at the "decision point" like the fridge door means the reminder fires exactly when the user is about to grab food, which makes the intervention more effective.

ESP32 is a family of low-cost, low-power Wi-Fi/Bluetooth dual-mode microcontroller chips from Espressif, typically clocked at 160–240MHz with built-in wireless connectivity, and dev boards usually cost anywhere from a dozen to a few dozen RMB. It's widely used for IoT prototypes and maker projects, with a mature ecosystem supporting Arduino, MicroPython, ESP-IDF, and other development frameworks. In this project, the ESP32 communicates with the phone app over Wi-Fi (e.g., via MQTT or HTTP interfaces) to fetch calorie data, then drives a small TFT/OLED screen to draw the colored rings. Limited by memory and compute, the ESP32 itself doesn't participate in model inference — it only serves as the "display terminal" receiving data and rendering the interface, which is exactly where the "phone computes, hardware displays" division of labor comes from.

Open source and reproducibility

The creator says the project is open-sourced in the ESP32 intake repository. This means others can reproduce it and build on the existing code — for example, adjusting the ring thresholds, changing the screen size, or integrating other local models.

The combination of on-device phone models plus low-cost hardware lowers the barrier for this kind of project. The ESP32 itself is affordable with a mature ecosystem, and running local models on the phone avoids extra compute device investment. For people who want to try small "AI + hardware" projects, this is a relatively lightweight entry point.

A few things worth noting

The accuracy of calorie estimation is limited by photo quality and food complexity. In Chinese cuisine scenarios with many mixed dishes and hard-to-distinguish ingredients, the estimates are better treated as a reference range than precise values. The creator demonstrated recognition results on cafeteria lunch in the video; real-world performance may vary across different scenarios.

Also, deploying the model on-device places certain demands on device performance, and the specific compatibility range should be checked against the project repository's documentation.

From a product perspective, the value of this project isn't in how complex the technology is, but in closing the loop of "recognize — estimate — remind" within a very natural life scenario. The phone handles computing, the ESP32 handles display, and the fridge door handles triggering — each playing its own role.

One-line Summary

A Bilibili creator open-sourced a closed-loop project combining on-device phone model photo calorie estimation with an ESP32 mini-screen tri-color ring reminder.

Paragraph Summary

Bilibili creator "电子博Studio" open-sourced a small project combining AI food recognition with a hardware reminder. The phone app uses the open-source miniCPM series model to perform food photo recognition and calorie estimation locally, with no need to upload images to the cloud, balancing privacy and low latency. A small ESP32 screen syncs app data and visually displays calorie intake status with green, yellow, and red rings; a magnet on the back lets it attach to the fridge door, next to the coffee machine, or in the snack area, triggering reminders at the moment users are about to eat. The project is open-sourced in the ESP32 intake repository, supporting reproduction and secondary development. The article also notes that calorie estimation is affected by photo quality and food complexity — in Chinese mixed-dish scenarios it's better treated as a reference range — and that on-device phone models have certain performance requirements.