Power law

What it is

A power law in machine learning describes a steady, predictable relationship between resource scaling and error reduction. As you multiply the compute budget or dataset size by a factor such as ten, the error rate drops by a fixed percentage. This means performance does not hit sudden walls or random plateaus until you reach the limits of current hardware or data availability. For example, moving from one billion parameters to ten billion parameters yields a measurable, consistent decrease in loss across orders of magnitude.

Why it matters

Knowing this relationship dictates your infrastructure budget, model selection strategy, and feature roadmap. If you are deciding whether to upgrade to a model that is ten times larger, power laws let you calculate the expected accuracy gain before spending capital on inference or fine-tuning. Ignoring these scaling trends leads to misallocated compute budgets, where you might overpay for marginal improvements or waste time trying to optimize architectures that are fundamentally limited by scale.

In practice

You observe power laws when comparing models of different parameter sizes on benchmarks or tracking how latency and cost scale with context length. When selecting an API tier or open weights model, you use these scaling curves to determine the minimum parameter count required to pass your accuracy threshold. You do not tune a power law directly via an API parameter, but you rely on it when choosing between smaller, cheaper models and massive frontier models.

The tradeoff

Scaling yields predictable gains, but the costs grow non-linearly while performance improvements follow diminishing returns at the extreme high end.

Where it appears

Research summaries that use Power law, each linked to its source paper.

• Explaining Neural Network Learning Dynamics
  Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

  The authors derive a mathematical framework that models how neural network training converges using a small set of variables regardless of the model size.

• Balancing AI Safety Methods at Scale
  Rules or Character? Scaling Laws for AI Safety Design

  The paper models how engineers should shift resources between training-time character shaping and inference-time rule enforcement as system deployment scale increases.

• Generalizing Transformer Attention with Power Laws
  Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

  The paper introduces Power Law Graph Attention as a flexible, learned alternative to the standard fixed-operator attention used in modern transformer models.