Key points
- A model's error falls on a predictable power-law curve as size, data and compute grow.
- Each 10x step costs ten times more than the last and buys a smaller gain.
- OpenAI forecast GPT-4's final loss from runs that used up to 10,000 times less compute.
- The curve held up; the advice on how to split a budget between size and data did not.
Error falls on a predictable curve
In 2020, an OpenAI team led by Jared Kaplan published "Scaling Laws for Neural Language Models." They trained more than 200 transformer language models, covering seven orders of magnitude in compute. Across that whole range, the model's error went down as a power law in model size, data, and compute.
The error they measured is called loss. It's roughly how surprised the model is by the next word in text it hasn't seen. A power law shows up as a straight line on a log-log chart, and a straight line is easy to extend. That's the whole trick.
Illustration only, not measured data. Fit a line through the small runs, then extend it to the size you plan to train.
What a 10x jump buys you
Kaplan's fit gave an exponent of about 0.076 for model size and 0.050 for compute. Plug those in and you get rough numbers:
Each step costs ten times more than the last and buys a smaller gain.
These numbers describe language-modeling loss. They don't tell you how a model will do on any one task.
GPT-4 was forecast early in its training run
The best public test of this came in the GPT-4 Technical Report in 2023. OpenAI trained small models the same way they'd train GPT-4, using up to 10,000 times less compute. They fit a curve to those runs and predicted GPT-4's final loss on their internal codebase, which wasn't in the training data. They made the prediction shortly after the big run started, without peeking at partial results, and it landed close to the real number.
They did the same for HumanEval, a Python coding benchmark, using models with up to 1,000 times less compute. Their reason was practical. A run that size can't be tuned by trial and error, so they needed to know ahead of time where it would end up.
The curve held up. The spending advice didn't.
OpenAI's 2020 paper got the shape of the curve right. Where it went wrong was how to split a budget. Kaplan's team said model size should grow much faster than data. For 10x more compute, they suggested about 5.5x more parameters and only about 1.8x more training tokens.
DeepMind's Chinchilla paper changed that in 2022. After training more than 400 models, they found size and data should grow at the same rate: double one, double the other. To prove it, they trained Chinchilla, a 70B model, on 1.4 trillion tokens with the same compute used for Gopher, a 280B model trained on about 300 billion tokens. Chinchilla beat Gopher across a wide range of tasks. That's where the rule of thumb of roughly 20 training tokens per parameter comes from.
| Same compute budget | Gopher | Chinchilla |
|---|---|---|
| Parameters | 280 billion | 70 billion |
| Training tokens | About 300 billion | 1.4 trillion |
| Tokens per parameter | About 1 | About 20 |
| Result | — | Better across a wide range of tasks |
Later work kept Kaplan's power-law form and replaced his allocation exponent of 0.73 with something close to 0.5.
Labs now train past the "optimal" point on purpose
Chinchilla's ratio minimizes training cost. It doesn't account for what it costs to run the model afterward.
Scaling laws are still how labs make that call. They forecast loss well. Specific skills are harder to call in advance, and none of this is a guarantee. But it's why a company can commit to a huge training budget and have a decent idea what it'll get.