Improving LLM Training with Ranking Rewards

Summary & Methodology Analysis

Generative Reward Models often struggle in reinforcement learning workflows because they output comparative rankings rather than the scalar scores that algorithms like GRPO or DAPO require. The Ranking-based Reward Construction (RRC) method resolves this by converting these comparative outputs into usable rewards. It implements two main strategies: self-competitive ranking, which calculates rewards based on pairwise win counts among generated responses, and anchor-guided ranking, which benchmarks responses against a fixed set of anchor samples. To maintain stability, the authors incorporate majority voting and conflict-aware ranking adjustments into the self-competitive process. These adjustments help mitigate issues where the model's confidence, represented by token probabilities, might otherwise result in noisy or uncalibrated reward scores. By integrating these construction strategies, the system enables standard reinforcement learning pipelines to leverage generative models effectively. The authors demonstrate that scaling the inference-time computation in the reward model and increasing the number of anchors directly improves RL performance. Despite this, the researchers note that raw preference token probabilities are often poorly calibrated for scalar scoring, as they may reflect the model's subjective judgment confidence rather than objective human preference alignment. Consequently, using these probabilities directly as rewards can lead to collapse, necessitating the structured ranking approach proposed in the study.