Verifying Step by Step Reasoning in LLMs

Summary & Methodology Analysis

State-of-the-art large language models regularly produce logical mistakes and hallucinations during complex multi-step reasoning tasks. This paper addresses the need to carefully compare outcome supervision, which provides feedback for final results, and process supervision, which provides feedback for each intermediate reasoning step. To build these systems, all large-scale models were finetuned from GPT-4 and pretrained on a dataset of roughly 1.5B math-relevant tokens called MathMix. A generator model was finetuned on MATH training problems to produce solutions in a newline-delimited step-by-step format, and process supervision data called PRM800K was collected from human data-labelers who assigned positive, negative, or neutral labels to each step in model-generated solutions.

The authors employed an active learning strategy during data collection by strategically selecting convincing wrong-answer solutions, which are solutions rated highly by the current best process-supervised reward model but having incorrect final answers, to surface to data-labelers. They iteratively retrained the process-supervised reward model using the latest collected data at several points during the data collection process. Outcome-supervised Reward Models were trained by uniformly sampling solutions from the generator and training them to predict solution correctness based on automatically checked final answers. Process-supervised Reward Models were trained to predict the correctness of each step after its last token, maximizing the log-likelihood of target tokens and supervising only up to the first incorrect step. The process-supervised reward model score for a solution was defined as the product of the correctness probabilities for each step, considering neutral steps as positive.

Despite these advances, several limitations remain. The training sets for the large-scale outcome-supervised reward model and process-supervised reward model were not directly comparable, making a direct comparison challenging. Automatic grading used for outcome supervision targets is not perfectly reliable, as false positive solutions with correct answers and incorrect reasoning can be misgraded. Due to high data collection costs, at-scale ablations of data collection decisions were not feasible. A preliminary investigation into iterative retraining of the process-supervised reward model selector during data collection led to instability that could not be diagnosed, and the generalizability of these results beyond the domain of math is unknown.