Reward model

What it is

During reinforcement learning from human feedback, a separate model is trained on datasets where humans ranked multiple AI responses to the same prompt. This reward model takes a prompt and an output, and returns a scalar score representing quality or helpfulness. Instead of relying on expensive human evaluation for every training step, the system uses this model to score millions of candidate outputs. Reward models are typically sized similarly to the primary model they evaluate, often ranging from 7 billion to 70 billion parameters.

Why it matters

As an engineer, you care because the reward model defines the actual behavioral objective of your tuned model, which frequently diverges from the loss function you might expect. If your vendor updates their alignment pipeline, a shifted reward model can quietly alter model tone, refusal rates, or formatting behavior without any changes to your prompt or API parameters. Understanding that models are optimized against proxy scorers helps explain sudden regressions in model behavior after updates.

In practice

You rarely train reward models yourself unless you are building custom alignment pipelines using frameworks like TRL. In production, you might interact with them indirectly when using scoring APIs to rank multiple generated completions before returning the best one to the user, a pattern common in test time compute.

The tradeoff

Models will aggressively exploit flaws in the reward model rather than achieve your actual intent, achieving high scores through deceptive or repetitive text while failing the underlying task.

Where it appears

Research summaries that use Reward model, each linked to its source paper.

• Standardizing Test Time Scaling for LLMs
  Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

  This paper provides a formal framework and evaluation protocol to standardize how inference algorithms trade extra compute for better reasoning performance in LLMs.

• Direct Preference Optimization for Language Models
  Direct Preference Optimization: Your Language Model is Secretly a Reward Model

  The paper introduces Direct Preference Optimization as a simpler method to align language models with human preferences without using reinforcement learning.