Finding the Optimal Size for Language Models
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- Current large language models are significantly undertrained for a given compute budget.
- The paper trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens.
- Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, representing a greater than 7% improvement over Gopher.
- Chinchilla achieves a perplexity of 7.16 on Wikitext-103 compared to 7.75 for Gopher.
- Chinchilla improves average BIG-bench performance by 10.7%, reaching an accuracy of 65.1% versus 54.4% for Gopher.
Summary & Methodology Analysis
To investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget, the authors trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. They used three distinct approaches for their analysis. First, they fixed model sizes and varied the number of training sequences and cosine cycle length to map FLOP count to training loss. Second, they used IsoFLOP profiles by varying model size for a fixed set of training FLOP counts and fitted a parabola to each curve. Third, they fitted a parametric loss function modeling final losses as a function of parameter count and number of seen tokens. These methods helped determine that current large language models are significantly undertrained.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem investigated in the paper?
The paper investigates the optimal model size and number of tokens for training a transformer language model under a given compute budget, finding that current large language models are significantly undertrained.
Q2. How many language models were trained for the study?
Over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens.
Q3. What is Chinchilla?
Chinchilla is one of the models evaluated in the paper, which achieves strong performance improvements over prior models like Gopher.
Q4. What was the first approach used in the methodology?
Fixing model sizes and varying the number of training sequences or cosine cycle length to map FLOP count to training loss.
Q5. What was the second approach used in the methodology?
Using IsoFLOP profiles by varying model size for a fixed set of training FLOP counts and fitting a parabola to each curve.
Q6. What was the third approach used in the methodology?
Fitting a parametric loss function modeling final losses as a function of parameter count and number of seen tokens.
Q7. How does Chinchilla perform on the MMLU benchmark compared to Gopher?
Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, representing a greater than 7% improvement over Gopher.
Q8. What are some limitations regarding scale mentioned in the paper?
Due to the cost of training large models, there are only two comparable training runs at large scale (Chinchilla and Gopher) without additional tests at intermediate scales.
Q9. What data duration limitation applies to the training runs?
All training runs for the analysis were trained on less than an epoch of data.