Optimizing Data Repetition in LLM Pretraining

Summary & Methodology Analysis

When scaling large language models, a primary challenge is the dilution of high-quality data as the training token budget grows relative to the model parameters. This paper investigates strategies to maintain performance by repeating specific domain data while keeping the tokens-per-parameter ratio (TPP) constant. The researchers systematically analyzed how varying the repetition count affects model outcomes across different scales of model sizes to ensure that resources are not wasted on low-utility training cycles. By fixing the TPP, the team isolated the effect of data repetition as a variable in the pretraining pipeline.

The methodology focuses on correlating the optimal repetition frequency with both the final validation loss of the domain and the quantity of unique data available. The analysis reveals that larger models permit a slightly higher repetition count for high-quality data, suggesting that capacity impacts how effectively a model can ingest repeated patterns. Furthermore, the results indicate that the inherent quality of the domain data, represented by lower validation loss, is a much stronger predictor of how much repetition a domain can sustain than the raw volume of unique data provided.

While repeating data is a viable strategy to maximize the utility of high-quality datasets, the paper identifies clear limitations. The primary risk is that excessive repetition leads the model to overfit, which degrades general performance. Because the paper does not specify precise parameter counts or exact training costs, practitioners should monitor their own validation metrics closely when applying these findings to their specific workloads to avoid the negative effects of over-saturation.