Temporal Variability Score
A metric measuring the consistency of a knowledge item across web text from different years, defined as the coefficient of variation of the item's negative log-likelihood across models at multiple time points. It is used to diagnose knowledge learning dynamics during pretraining.
Timeline (last 90 days)
The study proposes the temporal variability score (TVS), defined as the coefficient of variation of a knowledge item's negative log-likelihood across time-point models trained on web snapshots from multiple different years.
The study used six models from multiple open-source model families, covering benchmarks on six different topics, to evaluate the relationship between TVS and learning dynamics.
In the early stage of training, low-TVS and high-TVS items are learned at similar speeds; as training progresses, high-TVS items improve faster.
In 34 of the 36 model-benchmark settings, high-TVS items outperformed low-TVS items at the end of pretraining, with an average advantage of 7.3 percentage points in opportunity-normalized accuracy.
The pattern that high-TVS items perform better on knowledge-intensive tasks is the most robust, even when comparing items of similar difficulty.
One possible explanation is that high-TVS items correspond to knowledge that appears frequently in a specific time period but is not universally present throughout, and the model may better capture these time-varying knowledge signals in the later stage of pretraining.
When model capacity is limited, consistently common and consistent knowledge (low TVS) is compressed into representations first, while the time-fluctuating fine-grained signals carried by high-TVS items may require more parameters and longer training to be fully encoded.
Aggregated benchmark scores may mask different patterns of knowledge learning, and TVS can serve as a diagnostic tool to help understand how models absorb different types of knowledge during pretraining.
All Facts (8)
The study proposes the temporal variability score (TVS), defined as the coefficient of variation of a knowledge item's negative log-likelihood across time-point models trained on web snapshots from multiple different years.
50%UnverifiedThe study used six models from multiple open-source model families, covering benchmarks on six different topics, to evaluate the relationship between TVS and learning dynamics.
50%UnverifiedIn the early stage of training, low-TVS and high-TVS items are learned at similar speeds; as training progresses, high-TVS items improve faster.
50%UnverifiedIn 34 of the 36 model-benchmark settings, high-TVS items outperformed low-TVS items at the end of pretraining, with an average advantage of 7.3 percentage points in opportunity-normalized accuracy.
50%UnverifiedThe pattern that high-TVS items perform better on knowledge-intensive tasks is the most robust, even when comparing items of similar difficulty.
50%UnverifiedOne possible explanation is that high-TVS items correspond to knowledge that appears frequently in a specific time period but is not universally present throughout, and the model may better capture these time-varying knowledge signals in the later stage of pretraining.
50%UnverifiedWhen model capacity is limited, consistently common and consistent knowledge (low TVS) is compressed into representations first, while the time-fluctuating fine-grained signals carried by high-TVS items may require more parameters and longer training to be fully encoded.
50%UnverifiedAggregated benchmark scores may mask different patterns of knowledge learning, and TVS can serve as a diagnostic tool to help understand how models absorb different types of knowledge during pretraining.
50%