Kumo, Nori, Chronos-2 and LGBM: Electricity Price Forecasting Benchmark

When Foundation Models Meet Electricity Price Forecasting: A Cross-Model-Category Benchmark

Electricity price forecasting has long been one of the most challenging tasks in the time series domain—it is simultaneously shaped by weather, supply and demand, fuel costs, and even market sentiment, producing violent fluctuations and substantial non-stationarity. Traditionally, gradient boosted trees (such as LightGBM and CatBoost) have long held the mainstream position in such forecasting tasks thanks to their strong modeling capability for tabular features. Recently, two categories of foundation models have begun to enter this field: tabular foundation models (Kumo, Nori) and time series foundation models (Chronos-2, TimesFM-3).

A researcher shared a benchmark on Reddit targeting German day-ahead electricity price forecasting, systematically comparing the performance of the above models under "zero-shot" conditions and exploring the impact of additional training data on traditional ML baselines. The design philosophy and conclusions of this experiment offer reference value for practitioners interested in deploying foundation models on time series tasks.

Experimental Design: How to Fairly Compare Models Across Categories

The core challenge of this benchmark lies in the fact that foundation models and tree models follow fundamentally different usage paradigms. Tree models require feature engineering and hyperparameter tuning tailored to electricity price data, whereas foundation models perform direct inference after being pretrained on large-scale general data. The author emphasizes that all foundation models were not trained on electricity market data, i.e., they operate in a zero-shot or training-free state.

To ensure a fair comparison, the author ensured that all model categories had access to a comparable information set during evaluation. This point is particularly critical—if the tree models were fed exogenous variables that the foundation models could not access, the comparison would lose its meaning. The experiment covers the typical scenario of German day-ahead electricity prices; this market's data is publicly transparent with pronounced volatility characteristics, making it suitable as a benchmark.

The Unique Challenge of Tabular Foundation Models: Avoiding Data Leakage

When using tabular foundation models for forecasting, an easily overlooked pitfall is data leakage. Since these models accept inputs in "tabular" form, if features are constructed improperly—for example, by encoding future information or target leakage into features—the model will exhibit inflated accuracy on the evaluation set.

The author specifically discusses how to avoid data leakage when using Kumo and Nori. This involves temporal alignment of features, the construction of rolling windows, and how to ensure that every row of features seen by the foundation model contains only information available before the forecast time point. For practitioners looking to try tabular foundation models, this section offers strong practical guidance.

Data leakage is a systemic risk in time series forecasting, rooted in the inclusion of information in training data that would only be available after the forecast time point. Common forms include: using global statistics (such as full-period mean and variance) for standardization, incorrectly incorporating future target values into features during rolling window construction, or performing target encoding on categorical features without temporal splitting. For tabular foundation models, the risk is further amplified because such models typically receive features on a "row" basis and lack explicit awareness of temporal order. If a column in the feature matrix implicitly contains future information, the model will learn spurious strong correlations, leading to inflated accuracy on the evaluation set and rapid failure in real deployment. Preventive measures include strict temporal splitting, rolling-origin evaluation, and auditing the availability time point of each feature.

Tabular foundation models are a class of models pretrained on large volumes of heterogeneous tabular datasets, with representative works including Kumo and Nori. Unlike traditional tree models, they do not rely on manual feature engineering but instead learn inter-column relationships, cell semantics, and table structure to generate predictions. Their typical usage is to organize the samples to be predicted into a table, and the model performs direct inference based on the "tabular prior" acquired during pretraining. In time series forecasting, this means encoding lagged values, exogenous variables, timestamps, and so on as table columns. The appeal of this paradigm lies in its cross-task generalization potential, but the challenges are also prominent: tabular foundation models are sensitive to column semantics and data types, and improper temporal alignment can easily introduce leakage; meanwhile, their inference cost and interpretability are often less transparent than those of gradient boosted trees.

Comparison with Tuned Traditional Tree Models: Who Comes Out on Top

Another focus of the experiment is the comparison between foundation models and tuned LightGBM and CatBoost. The traditional ML baselines were not configured arbitrarily but underwent targeted hyperparameter tuning. This comparison design answers a practical question: without investing substantial feature engineering and tuning costs, can foundation models match or even surpass carefully tuned tree models?

According to the author's summary in the Reddit post, the full article covers the experimental setup, leak-free usage of tabular foundation models, comparison results with tuned tree models, and the impact of additional training data on traditional ML baselines. Although the post itself does not provide specific numerical conclusions, the comparison framework itself already reveals the positioning question for foundation models in real business scenarios—are they replacements, or rapid baselines?

Zero-shot and training-free are two key concepts in foundation model evaluation. Zero-shot refers to direct inference by a model on a completely unseen task or dataset without any parameter updates; training-free emphasizes that not even fine-tuning is performed, relying solely on the general representations acquired during pretraining. In time series scenarios, zero-shot inference means the model takes the electricity price series and its covariates as input and directly outputs future forecast values. The advantage of this paradigm is low deployment cost and fast iteration, but the disadvantages are also obvious: the model cannot adapt to the volatility patterns, holiday effects, or policy shocks of a specific market. Understanding the spectrum between zero-shot, few-shot, and fine-tuning helps determine the applicable boundaries of benchmark conclusions—poor zero-shot performance does not equate to foundation models being worthless on that task, but may merely indicate the necessity of fine-tuning or in-context learning.

The Impact of Additional Training Data on Traditional Models

The experiment also examines another dimension: how the performance of traditional ML baselines changes when they are provided with more training data. This design touches on the core value proposition of foundation models—they claim to have acquired generalization capability through large-scale pretraining, thereby reducing dependence on domain-labeled data. If tree models can significantly catch up as data volume increases, then the advantageous range of foundation models needs to be defined more precisely.

The answer to this question varies by task. For forecasting tasks with a high degree of structure and rich exogenous variables, such as German day-ahead electricity prices, tree models combined with good feature engineering tend to perform strongly. The value of foundation models may be more reflected in cold-start scenarios or data-scarce markets.

Practical Implications and Open Questions

This benchmark offers several noteworthy signals. First, evaluating foundation models in time series forecasting requires extra care regarding data leakage and feature alignment; otherwise, the conclusions are unreliable. Second, the comparison between zero-shot foundation models and tuned tree models should be regarded as a comparison of two workflows, rather than a pure comparison of model architectures. Third, the impact of additional data on traditional models determines the marginal value of foundation models in data-rich scenarios.

The author includes links to the full discussion and tutorial in the Reddit post, where readers can further examine the experimental setup and result details. For teams currently evaluating whether to introduce foundation models into electricity price forecasting or other energy time series tasks, this benchmark provides a rigorous starting point.

A note: this experiment focuses solely on the German day-ahead electricity price market, and extrapolating the conclusions to other markets or time scales requires caution. In addition, the inference cost, deployment complexity, and interpretability of foundation models are also factors that cannot be ignored in practical model selection.

Background Supplement

The German day-ahead electricity market is one of the core reference markets for European electricity trading, operated by EPEX Spot, which uses an hourly auction mechanism to form clearing prices for each hour of the next day. Its price series exhibits pronounced intraday double peaks, weekend effects, holiday jumps, and frequent negative prices, and is jointly driven by wind and solar output forecasts, unit maintenance, fuel costs, and cross-border transmission constraints. The difficulty of day-ahead price forecasting lies in the fact that forecast errors in exogenous variables propagate to price forecasts, and market coupling allows supply and demand changes in neighboring countries to permeate rapidly. This market's data is public, finely granular, and has a long history, and is therefore often used as an electricity price forecasting benchmark. However, it should be noted that Germany's renewable energy penetration rate and policy environment differ considerably from other regions, so extrapolating conclusions requires caution.

One-line Summary

A benchmark and workflow comparison of zero-shot foundation models versus tuned tree models for German day-ahead electricity price forecasting.

Paragraph Summary

A benchmark targeting German day-ahead electricity price forecasting systematically compares the tabular foundation models Kumo and Nori and the time series foundation models Chronos-2 and TimesFM-3 under zero-shot conditions, using hyperparameter-tuned LightGBM and CatBoost as traditional baselines. The experiment emphasizes that fair comparison requires ensuring all model categories access a comparable information set, and specifically discusses feature temporal alignment and data leakage prevention in the use of tabular foundation models. The article also examines the impact of additional training data on traditional ML baselines to define the marginal value of foundation models. The core conclusion is: the comparison between foundation models and tuned tree models should be viewed as a comparison of two workflows rather than a pure judgment of architectural superiority; the conclusions apply only to the German market and require caution when extrapolated, and inference cost and interpretability are also key factors in model selection.