Understanding Reinforcement Learning Post-Training

Summary & Methodology Analysis

This paper tackles two open questions regarding reinforcement learning post-training for large language models, specifically examining how pretraining choices shape returns to reinforcement learning compute and what reinforcement learning actually does to the model policy. To study this, the researchers pretrain language models ranging from 5M to 1B parameters on human chess games using an autoregressive next-token prediction objective, which is the standard method of predicting the next token in a sequence. Following pretraining, they perform supervised fine-tuning, an adaptation step using labeled data, on synthetic reasoning traces generated by sampling game continuations from the pretrained policy and structuring them into depth-first trees.

For the optimization phase, the models are trained via Group Relative Policy Optimization, an algorithm that optimizes policies relative to a group of sampled outputs, on a chess puzzle environment with verifiable binary rewards. The authors then analyze policy evolution by evaluating induced move policies, fitting power-sharpening transformations, and categorizing policy changes across puzzle difficulty bins. Their evaluation incorporates several models and datasets, including Lichess, Qwen3, OLMo-2, Nemotron-CC-Math-v1, Dolma3, NuminaMath-CoT, GSM8K, MATH, and DeepScaler.

However, the study has notable limitations. Chess differs from natural language in ways that limit direct transfer, such as having a small vocabulary, exact verification, and no entanglement with world knowledge or fluency. Furthermore, the models reach at most 1B parameters, meaning scaling trends may behave differently at larger scales. Finally, the reinforcement learning environment uses puzzles with unique designated solutions and binary rewards, which is a restricted form of verification compared to partial-credit or open-ended rewards in language tasks.