Power law
A mathematical relationship where model performance improves predictably as compute, data, or parameter counts multiply.
What it is
A power law in machine learning describes a steady, predictable relationship between resource scaling and error reduction. As you multiply the compute budget or dataset size by a factor such as ten, the error rate drops by a fixed percentage. This means performance does not hit sudden walls or random plateaus until you reach the limits of current hardware or data availability. For example, moving from one billion parameters to ten billion parameters yields a measurable, consistent decrease in loss across orders of magnitude.
Why it matters
Knowing this relationship dictates your infrastructure budget, model selection strategy, and feature roadmap. If you are deciding whether to upgrade to a model that is ten times larger, power laws let you calculate the expected accuracy gain before spending capital on inference or fine-tuning. Ignoring these scaling trends leads to misallocated compute budgets, where you might overpay for marginal improvements or waste time trying to optimize architectures that are fundamentally limited by scale.
In practice
You observe power laws when comparing models of different parameter sizes on benchmarks or tracking how latency and cost scale with context length. When selecting an API tier or open weights model, you use these scaling curves to determine the minimum parameter count required to pass your accuracy threshold. You do not tune a power law directly via an API parameter, but you rely on it when choosing between smaller, cheaper models and massive frontier models.
The tradeoff
Scaling yields predictable gains, but the costs grow non-linearly while performance improvements follow diminishing returns at the extreme high end.