What Improves During Pretraining? Temporal Variability Score Reveals Knowledge Learning Dynamics

What Improves During Pretraining? Temporal Variability Score Reveals Knowledge Learning Dynamics

Language models pretrained on web corpora collected over many years acquire both general and domain knowledge. But how this knowledge is learned during pretraining has remained unclear. A recent study uses point-in-time models trained on web snapshots from different years to propose the Temporal Variability Score (TVS), which quantifies the consistency of knowledge across web text from different years. The study finds that high-TVS and low-TVS entries learn at similar rates early in training, but high-TVS entries improve faster later on and ultimately perform better in most model-benchmark combinations.

What Is the Temporal Variability Score (TVS)?

TVS is defined as the coefficient of variation of an entry's negative log-likelihood across multiple point-in-time models. Simply put, if a fact appears frequently and is phrased consistently in web text across different years, its negative log-likelihood should be relatively stable across point-in-time models, yielding a low TVS; conversely, if a fact is only popular in specific years or changes greatly over time, different models will predict it with greater fluctuation, yielding a high TVS.

The study used six models from multiple open-source model families, covering benchmarks on six different topics, to evaluate the relationship between TVS and learning dynamics. TVS provides a diagnostic tool that goes beyond aggregate benchmark scores to reveal differences in how knowledge is learned during pretraining.

Negative log-likelihood (NLL) is one of the most fundamental loss metrics in language model pretraining and evaluation. For a knowledge entry (e.g., a fact or question-answer pair), the model generates a probability value representing how likely it considers the answer to be correct; NLL is the negative logarithm of that probability. The lower the NLL, the more confident and accurate the model's prediction for that entry. On point-in-time models, the same entry yields different NLL values across models trained on snapshots from different years. TVS uses the "coefficient of variation" (standard deviation divided by mean) rather than plain standard deviation because the coefficient of variation is a dimensionless normalized measure that eliminates the influence of differences in absolute NLL levels across entries, making comparisons across entries and models possible.

Chance-normalized accuracy is a commonly used corrected metric in evaluation, computed as (model accuracy − random-guess accuracy) / (1 − random-guess accuracy). Because different benchmarks have different numbers of options and random baselines, raw accuracy is difficult to compare directly across benchmarks; chance-normalized accuracy maps performance onto a unified scale of "how much improvement over random guessing," making cross-benchmark averaging and comparison more meaningful.

High-TVS vs. Low-TVS Entries: Differences in Learning Pace

The analysis finds that early in training, low-TVS and high-TVS entries are learned similarly. But as training progresses, high-TVS entries improve faster. By the end of pretraining, in 34 of 36 model-benchmark settings, high-TVS entries outperform low-TVS entries, with an average advantage of 7.3 percentage points in chance-normalized accuracy.

This pattern is most robust on knowledge-intensive tasks, even when comparing entries of similar difficulty. This means TVS captures factors in knowledge learning that go beyond difficulty.

Point-in-time models are the methodological cornerstone of this study. Researchers extract corpora from different time periods from web archives (such as Common Crawl snapshots from various years) and train models with identical architectures but different data cutoff years. In this way, each model "freezes" the distribution of web knowledge from a specific period, amounting to a series of knowledge snapshots at different time slices. By comparing these models' predictive performance on the same set of entries, researchers can observe "how knowledge changes over time" separately from "how the model changes during training." Note that point-in-time models differ from conventional checkpoints: the former varies the time range of training data, while the latter varies the number of training steps on the same dataset. This study combines both dimensions to characterize entries' learning trajectories during pretraining.

Why Do High-TVS Entries Improve Faster Later On?

One possible explanation is that high-TVS entries often correspond to knowledge that appears frequently during specific time periods but is not always universally present. Models may better capture these time-varying knowledge signals later in pretraining. In contrast, low-TVS entries may correspond to knowledge that is always common and that models master early, leaving limited room for later improvement.

Another explanation relates to model capacity and training dynamics: high-TVS entries may require more training steps to integrate information scattered across different time segments, whereas low-TVS entries are represented stably earlier.

Curriculum learning is a related concept in pretraining data construction, referring to arranging training data in some order of difficulty or informativeness to improve learning efficiency. The results of this study have potential implications for curriculum design: if high-TVS entries indeed require more training steps to integrate scattered temporal signals, then appropriately increasing exposure to such knowledge later in pretraining might yield additional gains. On the other hand, model capacity refers to the amount of information the model's parameters can hold; when capacity is limited, always-common and consistent knowledge (low TVS) is compressed and represented first, while the fine-grained, time-fluctuating signals carried by high-TVS entries may require more parameters and longer training to be fully encoded. This offers a capacity-allocation perspective for understanding the "later improvement."

Implications for Pretraining and Evaluation

This study suggests that aggregate benchmark scores may mask different patterns of knowledge learning. TVS can serve as a diagnostic tool to help understand how models absorb different types of knowledge during pretraining. For pretraining data construction and curriculum design, considering the temporal variability of knowledge may help optimize learning efficiency.

In addition, point-in-time models provide a way to probe how knowledge popularity changes over time, which may in the future be used to evaluate how well models grasp time-sensitive knowledge.

Summary

The study uses TVS to reveal dynamic differences in knowledge learning during pretraining: high-TVS entries improve faster later on and ultimately perform better. This finding offers a new perspective for understanding the pretraining process and demonstrates the potential of temporal variability as a diagnostic tool.