Reinforcement learning

What it is

An agent observes the state of an environment, selects an action based on a policy, and receives feedback in the form of a scalar reward. The system updates its weights to prioritize sequences of actions that lead to higher long-term rewards. Unlike supervised learning which maps inputs to ground-truth labels, this approach requires defining a reward function that codifies success. Training often requires millions of simulation steps, making it significantly more compute-intensive than fine-tuning existing models.

Why it matters

You should care because reinforcement learning is the primary mechanism for optimizing LLM performance beyond their base capabilities, particularly in domains where a clear metric of success exists. Misconfiguring reward signals can lead to catastrophic failure modes or brittle behaviors that are difficult to debug in production. If you are building features that rely on autonomous decision-making or specialized chat behaviors, understanding how these models were aligned is critical for predicting their performance under edge cases.

In practice

In production, you rarely implement reinforcement learning from scratch, but you interact with its results when configuring parameters like temperature or choosing a model fine-tuned via RLHF. If your service uses an LLM to play games, navigate codebases, or manage tool use, you are essentially relying on a policy optimized via these methods. You would monitor reward metrics or latency to determine if the agent's decision-making logic is deviating from expected performance.

The tradeoff

The primary tradeoff is between the desire for optimal long-term outcomes and the risk of reward hacking, where the agent exploits flaws in the reward function to gain points without fulfilling the intended goal.

Where it appears

Research summaries that use Reinforcement learning, each linked to its source paper.

• Scaling Robot Manipulation via Action Alignment
  JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment

  JoyAI-RA 0.5 enables scalable robot manipulation by aligning diverse data sources like human videos and simulation into a shared format for consistent learning.

• Improving 3D Scene Generation with Agents
  iARCS: Iterative Agentic RL for Controllable 3D Scene Generation

  The iARCS method uses an LLM agent to iteratively refine 3D scene generation through reinforcement learning, ensuring better adherence to physical constraints and task requirements.

• Improving LLM Training with Ranking Rewards
  RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

  The paper introduces Ranking-based Reward Construction to bridge the gap between generative reward models and reinforcement learning algorithms.

• OneEmo: Unified Emotion AI Reasoning Model
  OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

  OneEmo is a 4.5B parameter multimodal model that improves emotion perception and understanding by using a novel reinforcement learning framework and a human-in-the-loop reasoning dataset.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• Improving Temporal Reasoning in Multimodal Models
  ChronoVision: Temporal Reasoning via Latent State Reconstruction

  ChronoVision introduces a visual-focused training framework to help multimodal large language models track and reason about continuous changes in images.

• Auditing Visual Tool-Use in Multimodal Models
  The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

  The paper investigates why multimodal large language models see marginal or negative accuracy gains when using active visual operations like crop-and-zoom, using a causal graph and a three-level intervention protocol to analyze returned visual evidence.

• Improving Multimodal Retrieval with Reasoning
  Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

  UniME-R1 enhances multimodal retrieval performance by integrating retrieval-centric chain of thought reasoning to correct errors where standard embedders fail.

• Automated Synthesis of Terminal Agent Tasks
  Recursive Synthesis for Long-Horizon Terminal Tasks

  The authors introduce Recursive Synthesis (RST) to automatically generate verifiable, long-horizon terminal-based tasks, reducing the cost of training agent data to roughly 0.05 dollars per task.

• LLM Agents Learn from Environment Exploration
  State2State: Environment-Derived Mid-Training for LLM Agents

  This paper introduces State2State, a new method for training large language model (LLM) agents that allows them to learn from their own exploration of an environment, rather than relying solely on predefined tasks or human feedback.

• Improving Long Chain LLM Reasoning Skills
  Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

  The researchers introduce a framework to measure and train LLMs to better navigate complex tasks that require switching between different types of reasoning skills.

• Improving AI Search Agents with Answer Backtracking
  ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment

  The researchers developed ABSeeker, a method that improves long-horizon search agent performance by assigning fine-grained rewards to individual steps based on their contribution to the final answer.

• Unified Multimodal Models for Agentic Generation
  ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

  ToolArtist improves agentic image generation by integrating multi-step reasoning and tool use into a single unified multimodal model.

• Evolving Rubrics for Better Audio Reasoning
  Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

  The researchers improved audio-language model reasoning by replacing static rewards with dynamically evolving, audio-grounded rubrics during reinforcement learning.

• Reducing Overthinking in Reasoning Language Models
  Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking

  The paper introduces a method called DASH that improves reasoning accuracy by identifying and penalizing unproductive self-reflection patterns in language models without needing manual annotations.

• Specialized Healthcare AI Agent Training
  Cura 1T: Specialized Model for Agentic Healthcare

  The paper introduces a recursive self-improvement training loop to develop a healthcare-focused model capable of managing patient consultations and clinical tool integration simultaneously.

• Improving AI Tool Use via Hindsight Reasoning
  TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

  TurnSight enhances tool-integrated reasoning by replacing trajectory-level reinforcement learning with fine-grained, turn-level hindsight signals derived from a model's own interactions.

• Stopping Visual Forgetting in Multimodal Models
  Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning

  Remember-R1 uses reinforcement learning to force multimodal models to maintain focus on visual evidence throughout long reasoning sequences.

• Automated Vulnerability Localization Using Agentic Models
  Antares: Foundation Models for Agentic Vulnerability Localization

  Antares is a compact, high-efficiency model designed to search through entire code repositories to identify specific file-level vulnerabilities without needing expensive external search tools.

• Enabling Native Computer Use for Agents
  Qwen-CUA: Native Computer Use for (almost) Everything

  The paper introduces Qwen-CUA, a system that enables autonomous agents to operate desktop and web software natively by processing visual screenshots instead of relying on specialized APIs or accessibility metadata.

• Teaching Computer Agents New Skills Automatically
  Progressive Agent Skill Generation via Reinforcement Learning

  The researchers created a way for artificial intelligence agents to learn and improve their skills by testing small changes to their behavior using reinforcement learning.

• Helping AI Remember Long Conversations
  LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

  The researchers developed LiveMem, a new system that allows artificial intelligence to store and retrieve past information even after its active memory limit has been reached.

• Improving Long Search Tasks With Vision
  DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents

  The paper introduces a new framework called DeepVoyager-VL that allows artificial intelligence agents to actively use visual information throughout a search process to solve complex, multi-step problems.

• Building 3D Designs One Step at a Time
  CADENA: Stepwise CAD Reverse Engineering

  The paper introduces CADENA, a system that reconstructs 3D computer aided design models by building them sequentially while checking for geometry errors at each step.

• Faster Text Generation Through Parallel Processing
  DiffusionGemma Technical Report

  DiffusionGemma improves language model speed by generating large blocks of text simultaneously instead of writing one word at a time.

• Making Quantum Circuit Design More Efficient
  DreamQAS: Learning a Decision-Useful World Model for VQE-Efficient Quantum Architecture Search

  The researchers developed a new system that reduces the massive computational effort required to find effective quantum computer circuit designs by predicting results instead of testing every single option.

• Making AI Code Reading More Efficient
  CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

  The researchers developed a technique to shrink images of computer code so that AI models can process them faster and more accurately without losing important information.

• Teaching Robots Better Visual Control
  WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

  The paper introduces the World Critic Model, a technique that improves robotic learning by helping robot vision systems better understand temporal movement and visual history.

• Smarter Image Editing Through Verified AI Rewards
  Evaluation-Verification Reward for Consistent Multi-Reference Image Editing

  The paper introduces a multi-dimensional evaluation and verification reward method to improve visual consistency and harmony in multi-reference image editing.

• Training Computer Agents With Evolving Virtual Environments
  Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale

  The paper introduces Echoverse, a system that builds deep, evolving virtual applications and tasks to effectively train computer-use agents.

• Mastering the Game of Go with AI
  Mastering the game of Go with deep neural networks and tree search

  The paper introduces AlphaGo, a system that uses deep neural networks and tree search to solve the complex game of Go by learning from both human expert games and self-play.

• Training Harmless AI Using AI Feedback
  Constitutional AI: Harmlessness from AI Feedback

  The paper presents a method for training harmless AI assistants without relying on human labels to identify harmful outputs.

• Teaching AI With Human Feedback
  Deep reinforcement learning from human preferences

  The paper introduces a method to train reinforcement learning systems using human preferences instead of a traditional reward function.

• End to End Robot Control Learning
  End-to-End Training of Deep Visuomotor Policies

  The paper investigates if training perception and control systems together as a single unit performs better than building them as separate components.

• Building Autonomous Lifelong Learning Agents
  Voyager: An Open-Ended Embodied Agent with Large Language Models

  Voyager is an LLM-powered embodied agent that uses iterative code generation and a skill library to perform continuous autonomous learning in open-ended environments.

• Improving AI Decision Making Through Reasoning
  ReAct: Synergizing Reasoning and Acting in Language Models

  The ReAct framework enhances language model decision making by interleaving logical reasoning traces with external tool actions.

• Learning Atari Game Policies from Pixels
  Playing Atari with Deep Reinforcement Learning

  The researchers developed a system that learns to play Atari games by processing raw screen pixels directly through a neural network.