Scaling Laws

The Scaling Law proposed by OpenAI reveals that model performance improves in a predictable power-law fashion as the number of parameters, training data volume, and compute increase.

The Scaling Law was first systematically quantified by Kaplan et al. in 2020 in language model research, pointing out a power-law relationship between model performance and parameter count, data volume, and compute.

In 2020, OpenAI researchers Kaplan et al. published a paper that first systematically revealed the predictable power-law relationships between language model performance and model parameter count, training data volume, and computational budget (Scaling Law)

Scaling Law refers to the predictable and regular improvement in model capability as the number of parameters and the volume of training data increase.

Scaling Law research has found that model performance exhibits a smooth power-law relationship with the number of model parameters N, dataset size D, and training compute C

The scaling laws were systematically demonstrated in 2020 by OpenAI researcher Jared Kaplan and others, with the core finding that the cross-entropy loss of language models follows a power-law relationship with model parameter count, training data volume, and compute

In 2020, Kaplan et al. at OpenAI revealed that model performance follows predictable power-law relationships with parameter count, training data volume, and compute budget.

缩放定律（Scaling Laws）由Kaplan等人在2020年提出，后经Chinchilla论文修正

Factors such as data quality, training techniques and hyperparameters, architectural details, and the choice of evaluation benchmarks all affect the model's final performance, and each factor introduces its own uncertainty

The paper "Scaling Laws for Neural Language Models" by Kaplan et al. came from OpenAI and was the first to systematically describe the relationship between model performance and parameter count, data size, and compute using power laws.

The paper found that for a fixed compute budget, training a smaller model for longer tends to yield better results than training a larger model, which contradicts the Chinchilla law.

The neural network scaling laws proposed by Kaplan et al. in 2020 indicate that model performance has a power-law relationship with data volume, parameter count, and compute, where the model's cross-entropy loss L and parameter count N satisfy L ∝ N^(-α).

Scaling Law research found that model performance (measured by cross-entropy loss) has a predictable power-law relationship with model parameter count, dataset size, and computational cost.

The Scaling Law paper by Kaplan et al. in 2020 revealed that, under a fixed compute budget, there is an optimal ratio between model parameter count and the number of training tokens (the Chinchilla optimal).

Scaling Laws refer to the power-law relationship between model performance (such as loss function values) and model parameter count, data volume, and compute.

The concept of Scaling Laws was first systematically proposed by institutions such as OpenAI and DeepMind in research around 2020, with the core finding that model cross-entropy loss exhibits a smooth power-law relationship with parameter count, training data volume, and compute, and shows no obvious signs of saturation over a fairly wide range.

Some views hold that the dividends of training scaling are still not exhausted, that multimodal, synthetic data, and post-training still have plenty of room for improvement, and that it is too early to declare Scaling Laws dead.

Traditional Scaling Laws describe the power-law relationship between model capability and training parameter count, data volume, and compute, with the implicit premise that a model's capability is fixed once it is trained.

The industry has recently observed that benchmark improvements from scaling have noticeably diminished, and high-quality training text data is approaching the upper limit of what is obtainable from the web.

Scaling Laws describe the decrease in pre-training loss, not a linear improvement in all downstream capabilities; some capabilities, such as complex reasoning, exhibit emergent properties.