Scaling Laws
A set of empirical relationships proposed by Kaplan and colleagues at OpenAI in 2020, describing power-law dependencies between model performance and parameter count, training data volume, and compute. These findings motivated the development of large-scale models such as GPT-3 and GPT-4.
Core Facts
Timeline (last 90 days)
In work such as GPT-2 and GPT-3, OpenAI systematically validated scaling laws and used them as the basis for planning its model R&D roadmap.
Scaling laws can not only predict performance but also inversely guide the design of data synthesis pipelines.
Scaling laws were initially widely validated in language model research, referring to the predictable power-law relationship of model performance as model scale, data volume, and compute increase.
Scaling laws describe the empirical regularity that model loss decreases as a power law with parameter count, data volume, and compute.
Scaling Laws underpinned the industry consensus that "bigger models are necessarily stronger" and influenced the decision-making logic behind compute investment and model scale.
Scaling Laws refer to the power-law relationship between model performance (such as loss function values) and model parameter count, data volume, and compute.
Scaling Laws were first systematically summarized by OpenAI and other institutions around 2020.
DeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.
Diffusion models perform better in global consistency but have slower inference speeds, while autoregressive models generate content frame by frame or segment by segment.
All Facts (10)
The Scaling Law proposed by OpenAI reveals that model performance improves in a predictable power-law fashion as the number of parameters, training data volume, and compute increase.
90%UnverifiedDeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.
70%UnverifiedIn work such as GPT-2 and GPT-3, OpenAI systematically validated scaling laws and used them as the basis for planning its model R&D roadmap.
50%UnverifiedScaling laws were initially widely validated in language model research, referring to the predictable power-law relationship of model performance as model scale, data volume, and compute increase.
50%UnverifiedScaling laws can not only predict performance but also inversely guide the design of data synthesis pipelines.
50%UnverifiedScaling laws describe the empirical regularity that model loss decreases as a power law with parameter count, data volume, and compute.
50%UnverifiedScaling Laws refer to the power-law relationship between model performance (such as loss function values) and model parameter count, data volume, and compute.
50%UnverifiedScaling Laws underpinned the industry consensus that "bigger models are necessarily stronger" and influenced the decision-making logic behind compute investment and model scale.
50%UnverifiedScaling Laws were first systematically summarized by OpenAI and other institutions around 2020.
50%UnverifiedDiffusion models perform better in global consistency but have slower inference speeds, while autoregressive models generate content frame by frame or segment by segment.
50%