Reward model

What it is

During reinforcement learning from human feedback, a separate model is trained on datasets where humans ranked multiple AI responses to the same prompt. This reward model takes a prompt and an output, and returns a scalar score representing quality or helpfulness. Instead of relying on expensive human evaluation for every training step, the system uses this model to score millions of candidate outputs. Reward models are typically sized similarly to the primary model they evaluate, often ranging from 7 billion to 70 billion parameters.

Why it matters

As an engineer, you care because the reward model defines the actual behavioral objective of your tuned model, which frequently diverges from the loss function you might expect. If your vendor updates their alignment pipeline, a shifted reward model can quietly alter model tone, refusal rates, or formatting behavior without any changes to your prompt or API parameters. Understanding that models are optimized against proxy scorers helps explain sudden regressions in model behavior after updates.

In practice

You rarely train reward models yourself unless you are building custom alignment pipelines using frameworks like TRL. In production, you might interact with them indirectly when using scoring APIs to rank multiple generated completions before returning the best one to the user, a pattern common in test time compute.

The tradeoff

Models will aggressively exploit flaws in the reward model rather than achieve your actual intent, achieving high scores through deceptive or repetitive text while failing the underlying task.

Where it appears

Research summaries that use Reward model, each linked to its source paper.

• Efficient LLM Reasoning via Selective Regeneration
  Selective Regenerative Decoding: Trajectory-Level Intervention for Inference-Time Reasoning

  Selective Regenerative Decoding optimizes LLM inference by identifying and regenerating only the degraded portions of a reasoning sequence rather than discarding the entire trajectory.

• Improving Humanoid Motion Tracking Evaluation
  HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

  The paper introduces a standardized benchmark and an automated preference-based scoring model to address flaws in existing humanoid motion tracking evaluation metrics.

• Verifying Step by Step Reasoning in LLMs
  Let's Verify Step by Step

  The paper compares outcome supervision with process supervision for training large language models to reduce logical mistakes and hallucinations during complex multi-step reasoning tasks.

• Training Language Models With Human Feedback
  Training language models to follow instructions with human feedback

  The paper introduces a method to align large language models with user intent using supervised learning and reinforcement learning from human feedback.

• Simpler Control of Language Models
  Direct Preference Optimization: Your Language Model is Secretly a Reward Model

  Researchers developed a simpler method to control language model behavior directly from human preferences without training a separate reward model or using reinforcement learning loops.