Stop Forcing LLMs Into Time-Series Forecasting! NeurIPS Tests Show: Removing Them Works Better

The time-series forecasting field is undergoing a course correction. Over the past two years, stuffing large language models (LLMs) into time-series forecasting pipelines has become almost a default move—but a NeurIPS Spotlight paper, using 13 datasets and 26 comparative experiments, delivers the opposite conclusion: remove the LLM and results actually improve. According to a related explainer video on Bilibili, representative "LLM + time-series" works like TimeLM, OneFaceOut, and Cuff, after removing the LLM or replacing it with a single layer of ordinary attention, saw parameter counts plunge from over 6 billion to just over 500,000 and training time drop from 3,000 minutes to 2 minutes—without any drop in accuracy.

This result is no isolated case. The top-ranked models on current time-series leaderboards—Kronos-2, TimeSafe M2.5, and others—are all time-series foundation models, not schemes that force an LLM into the forecasting pipeline. One line from the video nails the weak spot of such work: if a reviewer asks "does performance drop when you remove the LM?" and you can't answer, the whole paper falls apart.

The Real Role of LLMs in Time-Series: Reading Text, Not Crunching Numbers

So is the LLM completely useless in time-series forecasting? Not exactly—the problem is using it in the wrong place. The strength of large models has never been fitting numerical curves; it's understanding textual semantics. According to a study involving Google's Time team cited in the video, researchers re-ran evaluations using real data paired with real text and found that earlier "LLMs crush everything" conclusions, built on synthetic text, were severely overstated.

The truly robust approach turns out to be quite plain: a time-series foundation model handles numerical forecasting, the LLM reads the relevant text, each produces its own version of the result, and finally you take a weighted average. This combination outperformed the strongest single time-series foundation model in experiments. The key is a clear division of labor—let the model good at numbers handle numbers, let the model good at language handle language, rather than forcing one model to do both.

Time Series Foundation Model refers to models pretrained specifically on large-scale time-series corpora, with typical representatives including TimesFM, Kronos, and TimeSafe. Unlike LLMs, which process discrete text token by token, these models typically use patch-based input—slicing continuous numerical sequences into segments before encoding—and output numerical forecasts for several future steps directly. Their advantage lies in having seen massive amounts of real sequences during pretraining, giving them zero-shot or few-shot forecasting capability; they also avoid the information loss that comes from serializing numbers into text sequences. Understanding this distinction is the prerequisite for deciding "should I use an LLM or a dedicated time-series model": the former excels at semantics, the latter at numerical patterns.

Three Paths: Pick Your Approach Based on the Data You Have

The video's author offers a very practical decision framework: which path to take depends on what type of data you have on hand.

Path One: Pure Numerical Data—Build the Simplest Baseline First

If all you have is pure numerical sequences like electricity load, distributed power, or traffic flow, don't rush to bring in a large model. Start by treating the simplest model in the paper as your baseline: normalization, patching, one layer of attention—just these three. The author calls it PiTen, and it outperforms most encoding methods on small datasets. The Raven and Patch embedding modules provided in the materials can be assembled directly into your own improved version.

The criterion for improvement is clear: as long as you consistently beat this simplest baseline, then swap in the LLM for a comparison group and present accuracy and training time side by side—you've preemptively answered the question reviewers most want to ask.

Ablation Study is a standard technique in machine learning papers for verifying the contribution of each component: remove or replace a module one at a time and observe the performance change to judge whether that module truly matters. "Remove the LLM and see if performance drops" in this article is a direct application of ablation thinking. A well-designed ablation should hold all other conditions constant and change only the component under test; if metrics barely move after removal, the component is redundant and the gain may come from other parts or from data leakage. For "LLM + time-series" work, lacking this control group often means you can't rule out the alternative explanation that "ordinary attention achieves the same result."

Path Two: You Have Text Records—Run Four Control Experiments

If you have equipment inspection logs, maintenance plans, scheduling records, policy news, or weather forecast text, these are your differentiation chips. The approach: TimesFM 2.5 is open-source and requires no training, so it directly produces one version of the numerical forecast; the LLM reads the text portion and produces one version of semantic adjustment; then run four control experiments—pure time-series, pure LLM, weighted fusion of the two, and the most critical group: deliberately misaligning and shuffling the text before fusion.

If accuracy collapses immediately after the text is shuffled, that proves external text information is doing the real work—not that the model happened to fit noise. This design has both engineering significance and experimental rigor, enough to support a complete top-conference framework.

Shuffled Text Control is a rigor check for multimodal fusion: keep the text's quantity, length, and distribution unchanged, randomly shuffle only the correspondence between text and time segments, then run the same fusion pipeline. If accuracy drops noticeably, the model is genuinely exploiting semantic alignment between text and numbers; if accuracy is barely affected, the gain likely comes from extra parameter capacity or a regularization effect rather than the text information itself. This design costs very little but effectively answers reviewers' doubts about "whether the gain source is real," and it deserves to be a default control group in multimodal time-series research.

Path Three: No Text but You Need a "Large Model" Highlight—Use a Dedicated Time-Series Foundation Model

If your scenario has no text but your paper still needs a "large model" element, don't stuff data into an LLM yourself. Use a foundation model trained specifically for time-series, such as Kronos-2, which outputs forecasts without any training. Use it as a baseline: if your method can beat it, that's more convincing than beating ten traditional old models—because it represents the current performance bar for time-series foundation models.

Takeaways for Researchers

The value of this framework isn't in rejecting LLMs but in redefining their scope of responsibility. Many past "LLM + time-series" works were essentially making a language model fit numerical patterns it isn't good at, with extremely high parameter counts and training costs—and an unclear source of gains. The experimental design of this NeurIPS Spotlight paper—remove a component and see if performance drops—is exactly the standard move for testing whether such work holds up.

For students currently working on time-series forecasting research, the pragmatic path is: first use the simplest baseline to confirm the problem itself is solvable, then decide based on data type whether to introduce a text modality, and finally use ablation experiments to prove where the gains come from. The era of blindly stacking large-model parameters—at least in the time-series forecasting direction—is passing.

One-line Summary

NeurIPS research confirms that removing the LLM actually improves time-series forecasting; choose among three paths based on your data type.

Paragraph Summary

The article centers on a NeurIPS Spotlight paper showing that in time-series forecasting, removing the LLM or replacing it with a single layer of attention cuts parameters from over 6 billion to just over 500,000 and training time from 3,000 minutes to 2 minutes—with no loss in accuracy. The current leaderboard leaders—Kronos-2, TimeSafe M2.5, and others—are all dedicated time-series foundation models. The article stresses that the LLM's true value lies in reading textual semantics, and that weighted fusion with a numerical foundation model can outperform the strongest single model. The author accordingly lays out three paths by data type: for pure numerical data, build the simplest baseline first; with text, run four control experiments including shuffled-text control; without text, use a dedicated time-series foundation model as the baseline—and advocates using ablation experiments to prove the source of gains.