[KongchangAI]
· 2 min read· 1,004 words

Solo Developer Trains Transformer to Predict Blood Glucose, Small Models Prove Competitive

Solo Developer Trains Transformer to Predict Blood Glucose, Small Models Prove Competitive

Developer uses a 31K-parameter small model with synthetic pre-training and LoRA fine-tuning for personalized 2-hour glucose prediction.

A developer trained an encoder-only Transformer with just 31,253 parameters, pre-trained on ~19,200 CGM-years of synthetic data, then fine-tuned via LoRA on their own 115 days of CGM records to predict glucose 2 hours ahead. Inputs include CGM time series plus insulin and carbohydrate action curves; outputs are glucose in Kovatchev risk space, with a loss combining DILATE, MSE, and pinball. The model supports missing-trajectory imputation, backtesting, and can predict time of day from glucose patterns. It still falls short of the author for 30-minute hypoglycemia prediction, but the methodology shows the feasibility of small models plus synthetic data plus personalized fine-tuning in health AI, and the code is open-sourced.

A developer shared their project on Reddit: training an encoder-only Transformer with just 31,000 parameters, pre-trained on synthetic data and then fine-tuned with personal continuous glucose monitoring (CGM) data, to predict blood glucose up to 2 hours into the future. This case demonstrates the feasibility of small models + synthetic data + personalized fine-tuning in the health management space.

Model Architecture and Training Strategy

The model is an encoder-only Transformer with 31,253 parameters. The author first pre-trained it on a pre-generated synthetic CGM dataset, then used LoRA (Low-Rank Adaptation) to fit the model to their own personal data.

Training data came from the author's 115 days of CGM records, of which 80 days were used to train the adapter and 35 days for testing. The pre-training phase consumed approximately 19,200 CGM-years of synthetic data—composed of 1 million simulated type 1 diabetes patients, each with 7 days of data.

This "synthetic pre-training + personal fine-tuning" approach allows an extremely small model to capture general patterns of glucose variation, and then adapt to individual differences with a small amount of personal data. The author states that the model is competitive among published models.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique whose core idea is to inject low-rank decomposition matrices alongside the weight matrices of a pre-trained model, updating only these newly added small parameters during training while freezing the original model weights. In this scenario, it means the author doesn't need to retrain the entire 31K-parameter model—only a very small number of adapter parameters need to be trained to adapt general glucose patterns to personal physiological characteristics, saving computation and reducing overfitting risk. LoRA originally became popular in large language models, but its principle applies to any Transformer, and is especially suited to personalized health modeling with scarce data.

Input/Output Design: Action Curves and Risk Space

In addition to time-series CGM inputs, the model receives two additional inputs: insulin (U) and carbohydrates (G, GI), provided in the form of action curves. U is the exact number of insulin units, while (G, GI) is the author's best estimate.

The model outputs glucose values in the Kovatchev risk space, and the training loss combines DILATE, MSE, and pinball losses, weighted using learned Kendall-Gal weights. All models shown in the figures were trained with DILATE.

This design allows the model not only to predict glucose but also to perform counterfactual reasoning given future insulin and carbohydrate intake. The author notes that while there is no downstream benchmark to test this capability, in phone use the model does respond sensibly when accounting for future insulin and carbohydrates.

Kovatchev risk space is a mathematical transformation used in diabetes research to quantify the risk of glucose fluctuations. It maps glucose values into a nonlinear space where extremes of hypoglycemia and hyperglycemia are amplified, thereby more sensitively reflecting clinical risk. The specific transformation formula is: f(BG)=1.509×[ln(BG)^1.084−5.381], where BG is in mg/dL. In this space, risk values are symmetrically distributed around 0, making it easier for the model to learn and to unify loss weights across different glucose ranges. The model in this article directly outputs glucose values in risk space, meaning the prediction target naturally focuses on clinically dangerous regions rather than simply minimizing absolute error.

DILATE loss is a loss function designed specifically for time-series forecasting, short for "Distortion Loss including Shape and Time." It combines a shape loss (Dynamic Time Warping, DTW) and a temporal loss (TDI), penalizing deviations in both shape and timing of the predicted sequence. Compared with traditional MSE, DILATE is better at capturing sharp changes and delays in glucose curves, and is especially suited to critical patterns such as hypoglycemic events. This article combines it with MSE and pinball loss, and uses Kendall-Gal weights to learn the optimal balance of multi-task losses, avoiding manual tuning.

Features and Limitations

The model supports 2-hour-ahead prediction, missing trajectory imputation, and backtesting (predicting into the past). The native context window is variable from 3.5 to 7.0 days.

An interesting design choice: the model is trained to predict the time of day from past glucose, rather than taking time as an input feature. This capability only emerges when the context window reaches 48 hours or more.

Limitations are also clear: in predicting hypoglycemia about 30 minutes ahead, the model is still worse than the author themselves. This shows that at critical moments, human experience and intuition still have irreplaceable value.

Implications for Personal Health AI

The most noteworthy aspect of this project is its methodology: pre-train on large-scale synthetic data, then fine-tune with a small amount of personal data via LoRA, and you get a personalized small model. 31K parameters run effortlessly on a phone, yet achieve competitive predictive performance.

The source code is open-sourced on GitHub (0xdeadf1sh/T1DMAI), and developers interested in CGM data analysis and personalized health models can study it further. The design of the synthetic data simulator also provides a foundation for counterfactual reasoning, although standard evaluation benchmarks are still lacking.

For people with type 1 diabetes, such tools may in the future become an aid for managing blood glucose, but the author also cautions that the model still requires careful handling in critical scenarios such as hypoglycemia warnings.

One-line Summary

Developer uses a 31K-parameter small model with synthetic-data pre-training and LoRA fine-tuning to achieve personalized 2-hour glucose prediction.

Paragraph Summary

A developer trained an encoder-only Transformer with just 31,253 parameters, pre-trained on ~19,200 CGM-years of synthetic data, then fine-tuned via LoRA on their own 115 days of CGM records to predict glucose 2 hours ahead. Inputs include CGM time series plus insulin and carbohydrate action curves; outputs are glucose values in Kovatchev risk space, with a loss combining DILATE, MSE, and pinball. The model supports missing-trajectory imputation, backtesting, and can predict time of day from glucose patterns. It still falls short of the author for 30-minute hypoglycemia prediction, but the methodology demonstrates the feasibility of small models plus synthetic data plus personalized fine-tuning in health AI, and the code is open-sourced.

#Transformer#blood glucose prediction#CGM#LoRA#synthetic data#personalized health AI
Share:

Related articles