Chinchilla
A research paper published by DeepMind in 2022 that revised OpenAI's scaling laws by establishing the optimal ratio between model parameter count and training data volume for a given compute budget. Its findings have had a broad influence on large language model training strategies.
Core Facts
The Chinchilla law proposed by DeepMind further refined the optimal ratio relationship between data and parameters.
In 2022, DeepMind's Chinchilla paper pointed out that under the same compute budget, model size and training data volume should be scaled up proportionally.
DeepMind's Chinchilla research (2022) pointed out that optimal training should keep data volume and parameter count growing at roughly a 1:1 ratio
Timeline (last 90 days)
Work such as Chinchilla points out that under a given compute budget, parameter count and training token count should grow roughly in proportion to achieve optimal efficiency.
Chinchilla is a representative work in the field of scaling laws.
The conclusion of the Chinchilla scaling law: under a fixed compute budget, model parameter count and training token count should scale roughly in a 1:20 ratio in tandem, and early models were generally data-hungry.
The paper attributes the gap of Manacá-1B-Instruct-2 in general capabilities to pretraining scale (a comparison of compute budgets anchored on Chinchilla) rather than to the alignment method itself.
Chinchilla's experimental scale was limited, and its conclusions are based on specific datasets and training setups; whether they remain applicable to very large-scale models or domain-specific applications is a matter of debate.
The conclusions of the Chinchilla scaling law challenged the prevailing practice of the GPT-3 era, in which the industry invested most compute into scaling model size while training data was relatively insufficient.
DeepMind's Chinchilla paper and subsequent research on inference-time compute scaling show that dynamically increasing computation during the inference phase can significantly improve the output quality of smaller models.
The rule of thumb given by the Chinchilla scaling law is that the number of training tokens should be approximately 20 times the number of parameters
The Chinchilla scaling law originates from DeepMind's 2022 paper "Training Compute-Optimal Large Language Models"
The LLaMA series adopts an over-training strategy, training a 7B-parameter model on 1T tokens, far exceeding the Chinchilla-optimal ratio of 140B tokens
13 more timeline events
All Facts (20)
The Chinchilla law proposed by DeepMind further refined the optimal ratio relationship between data and parameters.
85%VerifiedIn 2022, DeepMind's Chinchilla paper pointed out that under the same compute budget, model size and training data volume should be scaled up proportionally.
85%VerifiedDeepMind's Chinchilla research (2022) pointed out that optimal training should keep data volume and parameter count growing at roughly a 1:1 ratio
80%VerifiedThe Chinchilla law states that under a fixed compute budget, appropriately reducing model size and increasing training data volume often yields better performance.
80%VerifiedGoogle DeepMind's Chinchilla paper optimized the optimal ratio between model parameter count and training data volume.
75%VerifiedThe emergent ability phenomenon was systematically documented by DeepMind researchers in the 2022 paper 'Emergent Abilities of Large Language Models'.
75%Unverified缩放定律（Scaling Laws）由Kaplan等人在2020年提出，后经Chinchilla论文修正
90%UnverifiedDeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.
70%UnverifiedChinchilla surpassed Gopher, which has 280 billion parameters, on multiple benchmarks with only 70 billion parameters.
70%UnverifiedThe core finding of the Chinchilla scaling law is that under a fixed compute budget, simultaneously increasing model parameters and training data volume is required to achieve optimal performance.
60%UnverifiedThe Chinchilla optimal scaling law was proposed by DeepMind in 2022, with the core formula C≈6ND, and the condition for optimal training being D≈20N.
60%UnverifiedThe Chinchilla model, with 70B parameters and 1.4T tokens, significantly outperforms the Gopher model, which has 280B parameters and 300B tokens, under the same compute budget.
60%UnverifiedChinchilla is a representative work in the field of scaling laws.
50%UnverifiedWork such as Chinchilla points out that under a given compute budget, parameter count and training token count should grow roughly in proportion to achieve optimal efficiency.
50%UnverifiedThe conclusion of the Chinchilla scaling law: under a fixed compute budget, model parameter count and training token count should scale roughly in a 1:20 ratio in tandem, and early models were generally data-hungry.
50%UnverifiedThe paper attributes the gap of Manacá-1B-Instruct-2 in general capabilities to pretraining scale (a comparison of compute budgets anchored on Chinchilla) rather than to the alignment method itself.
50%UnverifiedChinchilla's experimental scale was limited, and its conclusions are based on specific datasets and training setups; whether they remain applicable to very large-scale models or domain-specific applications is a matter of debate.
50%UnverifiedThe conclusions of the Chinchilla scaling law challenged the prevailing practice of the GPT-3 era, in which the industry invested most compute into scaling model size while training data was relatively insufficient.
50%UnverifiedDeepMind's Chinchilla paper and subsequent research on inference-time compute scaling show that dynamically increasing computation during the inference phase can significantly improve the output quality of smaller models.
50%UnverifiedThe Chinchilla scaling law originates from DeepMind's 2022 paper "Training Compute-Optimal Large Language Models"
50%