Back to Feed
Safety & Alignment

Training Language Models With Human Feedback

Original: Training language models to follow instructions with human feedback
By Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe Published 2022-03-04

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 8 concepts

Key Takeaways

  • InstructGPT models generate truthful and informative answers about twice as often as GPT-3 on the TruthfulQA benchmark.
  • On closed-domain tasks from the API prompt distribution, InstructGPT hallucinated information at a rate of 21 percent compared to 41 percent for GPT-3.
  • InstructGPT generated about 25 percent fewer toxic outputs than GPT-3 when prompted to be respectful on the RealToxicityPrompts dataset.
  • Outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3 in human evaluations.

Summary & Methodology Analysis

Large language models often express unintended behaviors such as making up facts, generating toxic text, or failing to follow user instructions because their next-token prediction training objective is misaligned with the goal of following user intent safely and helpfully. To solve this, the authors collected a dataset of labeler demonstrations of desired model behavior on prompts submitted through the OpenAI API and labeler-written prompts. They fine-tuned GPT-3 using supervised learning on this demonstration dataset to create supervised learning baselines. Next, they collected a dataset of human-labeled rankings and comparisons of model outputs on a larger set of API prompts to train a reward model that predicts which model output human labelers would prefer.

The supervised policy was then fine-tuned using the Proximal Policy Optimization algorithm to maximize the scalar reward output from the reward model, incorporating a per-token KL penalty from the supervised fine-tuning model and optionally mixing in pretraining gradients. This reinforcement learning from human feedback approach successfully steers the model toward helpful and safe behaviors without requiring changes to the underlying transformer architecture of GPT-3 itself, operating instead through specialized training phases that shape the output distribution.

Despite these improvements, the alignment procedure has notable limitations. InstructGPT still makes simple mistakes such as failing to follow instructions, making up facts, giving long hedging answers, or failing to detect instructions with false premises. Furthermore, the alignment procedure incurs an alignment tax resulting in performance regressions on certain public NLP datasets like SQuAD, DROP, HellaSwag, and WMT 2015 French to English translation, and does not significantly improve over GPT-3 on the Winogender and CrowSPairs datasets regarding bias.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the main problem addressed by the paper?

Large language models often express unintended behaviors such as making up facts, generating toxic text, or failing to follow user instructions because their next-token prediction training objective is misaligned with user intent.

Q2. What is InstructGPT?

InstructGPT is a series of models fine-tuned using supervised learning and reinforcement learning from human feedback to better follow user instructions.

Q3. Did the smaller InstructGPT model outperform larger GPT-3 models in human evaluations?

Yes, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3 in human evaluations.

Q4. How was the initial demonstration dataset collected?

The paper collected a dataset of labeler demonstrations of desired model behavior on prompts submitted through the OpenAI API and labeler-written prompts.

Q5. What role does the reward model play in the training pipeline?

A reward model is trained on a dataset of human-labeled rankings and comparisons to predict which model output human labelers would prefer.

Q6. Which reinforcement learning algorithm was used for fine-tuning the policy?

The supervised policy was fine-tuned using the Proximal Policy Optimization algorithm to maximize the scalar reward output from the reward model.

Q7. What specific penalty is incorporated during the reinforcement learning phase?

The training incorporates a per-token KL penalty from the supervised fine-tuning model and optionally mixes in pretraining gradients.

Q8. What performance trade-offs did the alignment procedure cause?

The alignment procedure incurs an alignment tax resulting in performance regressions on certain public NLP datasets like SQuAD, DROP, HellaSwag, and WMT 2015 French to English translation.

Q9. Did InstructGPT show improvement on bias datasets like Winogender and CrowSPairs?

The paper does not specify any significant improvement over GPT-3 on the Winogender and CrowSPairs datasets regarding bias.

Flag an issue

What is wrong with this summary?

Browse the archive →
Agents / Safety & Alignment Decoupling Persona and Execution for Agents The paper introduces a dual-domain architectural pattern that separates an AI agent's persona from its execution logic to improve governance and auditability in regulated environments. Benchmarks & Evals / Safety & Alignment Auditing Medical AI Reasoning Reliability Researchers evaluated how medical Large Language Models reason by systematically perturbing inputs and measuring if the model's logical chain stays consistent with its final output. Agents / Safety & Alignment Preserving Critical Constraints in LLM Workflows The paper demonstrates that standard LLM agent handoff processes often cause binding constraints to lose their functional power, and identifies methods to restore this operational state.