Reinforcement learning

What it is

An agent observes the state of an environment, selects an action based on a policy, and receives feedback in the form of a scalar reward. The system updates its weights to prioritize sequences of actions that lead to higher long-term rewards. Unlike supervised learning which maps inputs to ground-truth labels, this approach requires defining a reward function that codifies success. Training often requires millions of simulation steps, making it significantly more compute-intensive than fine-tuning existing models.

Why it matters

You should care because reinforcement learning is the primary mechanism for optimizing LLM performance beyond their base capabilities, particularly in domains where a clear metric of success exists. Misconfiguring reward signals can lead to catastrophic failure modes or brittle behaviors that are difficult to debug in production. If you are building features that rely on autonomous decision-making or specialized chat behaviors, understanding how these models were aligned is critical for predicting their performance under edge cases.

In practice

In production, you rarely implement reinforcement learning from scratch, but you interact with its results when configuring parameters like temperature or choosing a model fine-tuned via RLHF. If your service uses an LLM to play games, navigate codebases, or manage tool use, you are essentially relying on a policy optimized via these methods. You would monitor reward metrics or latency to determine if the agent's decision-making logic is deviating from expected performance.

The tradeoff

The primary tradeoff is between the desire for optimal long-term outcomes and the risk of reward hacking, where the agent exploits flaws in the reward function to gain points without fulfilling the intended goal.

Where it appears

Research summaries that use Reinforcement learning, each linked to its source paper.

• Using Language Models to Guide Robots
  $R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning

  The paper introduces a method that uses vision-language models to perform reasoning that guides robot manipulation policies, improving performance on long-horizon tasks.

• Agentic Multimodal Instruction Data Synthesis
  VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

  The researchers developed an agentic, iterative framework called VISA to generate high-quality training data for multimodal models by using feedback-driven loops instead of static one-pass pipelines.

• Faster LLM Reasoning via Prefix Sliding
  Prefix Sliding for efficient test-time scaling

  Prefix Sliding enables large language models to perform reasoning tasks three times faster without requiring additional training.

• Improving Visual Reasoning Through Scalable Training
  VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

  The authors introduce VBVR-Pro, a suite designed to improve visual reasoning capabilities in models by using verifiable generative tasks.

• Optimizing Reinforcement Learning for Faster Training
  WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

  WarpSAC is a scalable reinforcement learning framework that adapts its architecture based on available compute resources to accelerate training and improve deployment success.

• Improving AI Code Generation With Robust Testing
  Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

  The paper introduces a framework called RobustTests that improves AI code generation by synthesizing diverse, failure-inducing test cases to guide reinforcement learning.

• Improving AI Reasoning Through Verifiable Distillation
  On-policy Distillation with Verifiable Reward

  The paper introduces a method called OPDVR that aligns reinforcement learning signal with task success during model distillation to improve reasoning performance.

• Improving AI Search Agents With Co-evolving Feedback
  CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

  The paper introduces CAFE, a system that improves search agent performance and reduces hallucinations by alternatingly updating the agent and its critic through co-evolving feedback.

• Jointly Training AI to Create and Use Tools
  Joint Optimization of Tool Creation and Use for Large Language Model Agents

  The SMITH framework uses reinforcement learning to train a single language model to both synthesize reusable Python tools and apply them effectively to solve complex tasks.

• Improving Stable Critic Training for LLMs
  How to Train a Critic Stably and Efficiently

  The paper introduces BPCO, a method that stabilizes critic-based reinforcement learning, improving performance across various model sizes and tasks.

• Gaussian Guidance for Better Agent Training
  Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

  Agent-G2 optimizes agent training by using Gaussian-based guidance to sample expert trajectory lengths, achieving higher success rates at a fraction of the cost of traditional probing methods.

• Scaling Agentic Intelligence for Complex Work
  Apodex 1.1: Scaling Agentic Intelligence for Complex Work

  Apodex 1.1 provides a general purpose agentic system that scales intelligence for complex professional tasks across finance and science using a robust execution framework.

• Efficient Reinforcement Learning for Video Models
  Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

  The researchers introduced a method called OraRL that integrates ground truth annotations as oracle rollouts to improve video model performance and reduce inference latency.

• Simulating Realistic Human Shopping Behavior
  Towards Faithful Simulation of Human Shopping Behavior

  The paper introduces a GUI-grounded simulation agent that uses pixel-level perception and reinforcement learning to generate authentic, multi-turn e-commerce shopping trajectories.

• Improving AI Tool Use Through Mid-Training
  MidTool: Mid-training Data Synthesis for Agentic Tool Use

  The researchers created a 20.3B-token corpus called MidTool-Mix to improve agentic tool-use capabilities in models during the mid-training phase rather than relying solely on post-training.

• Self-Evolving Training Environments for Language Agents
  SPADE: Self-Play in Adaptive Synthetic Executable Environments

  SPADE improves agent performance by using an automated system that co-evolves training environments and reasoning agents through a continuous reinforcement learning loop.

• Controlling LLM Reasoning Costs via Self-Reflection
  Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models

  The paper introduces a training-free inference-time protocol that uses a self-critique loop and a confirmed sentinel to improve reasoning accuracy while early-stopping redundant computations.

• Improving Skill Selection for AI Agents
  SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

  SkillGate optimizes agent performance by partitioning training signals to stop task outcomes from interfering with how agents select procedural skills.

• Safely Shaping Agent Rewards with LLMs
  Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

  The paper demonstrates that using LLM progress scores to shape rewards for reinforcement learning agents does not alter the underlying optimal policy of the agent.

• Improving Multi-Reward Language Model Training
  Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

  The paper introduces SA-MRPO, a method that dynamically reweights reward objectives during reinforcement learning to prioritize underperforming metrics instead of treating all goals as equally weighted.

• Optimizing Reinforcement Learning for Autonomous Agents
  Agent Lightning v1.0: Towards Harnessed Agentic RL

  Agent Lightning v1.0 provides a declarative framework to manage the complex training loops required for agents that interact with external environments.

• Fine-Tuning LLM Agents with Less Memory
  Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

  The paper introduces Agentic ESOpt, a method for fine-tuning LLM agents that replaces traditional backpropagation with population-based parameter perturbations to reduce memory requirements during training.

• Teaching LLM Agents to Explore Better
  Clearing the Fog: Towards Installing and Refining Proactive Exploration Capabilities in LLM Agents

  The paper introduces a method using synthetic data and reinforcement learning to help language model agents explore proactive action strategies instead of falling into rigid behavioral patterns.

• Optimizing Vision Language Model Training Efficiency
  Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training

  Rollplex increases GPU utilization during vision-language model post-training by overlapping prompt processing with rollout decoding to eliminate serial execution bottlenecks.

• Improving Autonomous Driving with Failure-Informed Training
  FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving

  FIRE-VLA improves autonomous driving models by using a distillation method to correct consistent failures where reinforcement learning signals are insufficient.

• Unlocking Enhanced Spatial Reasoning for VisionLanguage Models
  SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

  The paper introduces SCOUT, a method combining structured chain-of-thought prompting and multi-objective reinforcement learning to fix spatial reasoning bottlenecks in vision-language models.

• Improving AI User Simulation for Better RL
  One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

  This paper addresses simulator collapse in multi-agent reinforcement learning by introducing techniques to diversify simulator responses and jointly train policies with multiple simulators.

• Automated Cloud Threat Detection Using Reinforcement Learning
  Machine Learning-Based Cyber Defense for Cloud Infrastructure: An Adaptive Deep Q-Network Architecture for Intelligent Intrusion Detection and Automated Threat Mitigation

  The authors implement a Deep Q-Network to provide real-time, autonomous intrusion detection and threat mitigation for cloud infrastructure.

• Improving AI Video Detection with Meta-Detection
  VidForensics-M1: Meta-Detection Reinforcement Learning with Verifiable Temporal Grounding for AI-Generated Video Forensics

  The researchers developed a reinforcement learning approach that uses verifiable temporal grounding to improve the accuracy of detecting AI-generated video forgeries.

• Agentic Visual Search for Long Documents
  InSight-doc: Agentic Visual Perception for Long-Document Understanding

  InSight-doc uses an agentic system that zooms into document regions to reduce computational overhead and hallucination in multimodal models.

• Building Systems for Adaptive Agent Co-Evolution
  Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design

  The paper presents a three-stage taxonomy to classify and structure how multiple agents and their environments can iteratively adapt to one another beyond static, single-entity learning models.

• Improving GUI Agent Accuracy Through Reflection
  LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

  LookAgain introduces a multi-turn refinement process that allows GUI agents to visually verify and adjust their coordinate predictions, significantly boosting accuracy on complex interfaces.

• Tool Calls Function Without Image Feedback
  Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

  Researchers discovered that tool use in vision language models relies on structured text scaffolding rather than the actual images returned by those tools.

• Controlling User Simulator Intent in Conversations
  Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

  The researchers developed a framework called UserIDA that allows developers to precisely control the local conversational intent of LLM-based user simulators.

• Robotic Foundation Model for Goal Navigation
  RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

  RynnValue is an open-source foundation model that improves robotic task success rates by predicting the remaining distance to a goal using visual observations.

• Building Open Models for Continual Learning
  Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA

  Macaron-V1 introduces a framework for deploying persistent agent models that update themselves through specialized adapters and recursive self-improvement loops.

• Improving Video Captioning with Atomic Rewards
  AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

  The paper introduces a new framework for training video captioning models that uses fine-grained, atomic-level rewards to improve accuracy in describing video content.

• Multi-Agent System for Deepfake Detection
  Multi-Agent Forensic Reasoning for Generalizable Deepfake Video Detection

  The paper introduces a multi-agent framework that uses specialized observation experts and reinforcement learning to identify forgeries in video content.

• Improving Agent Feedback with Trajectory Distillation
  Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

  The paper introduces a method to improve agent performance by dynamically allocating hindsight feedback across individual decision steps in a multi-turn task.

• Improving Japanese Structured Document Parsing
  Stockmark-Nemotron-3-Nano-Omni-JapanDocReader: Structured Document Parsing via Capability Injection and Forgetting Control

  The researchers developed Stockmark-Nemotron-3-Nano-Omni-JapanDocReader to balance structured document parsing with existing document visual question answering capabilities.

• Improving LLM Reliability for Scientific Data Analysis
  Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

  Fisher-R1 is a specialized LLM agent trained to perform reliable hypothesis testing by using a new benchmark and outcome-grounded reinforcement learning.

• Improving Mobile GUI Agents with Hindsight
  The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

  The researchers introduce Gated Hindsight Distillation to help GUI agents learn from future screenshots when current screen data is insufficient for decision making.

• Scaling Robot Manipulation via Action Alignment
  JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment

  JoyAI-RA 0.5 enables scalable robot manipulation by aligning diverse data sources like human videos and simulation into a shared format for consistent learning.

• Improving 3D Scene Generation with Agents
  iARCS: Iterative Agentic RL for Controllable 3D Scene Generation

  The iARCS method uses an LLM agent to iteratively refine 3D scene generation through reinforcement learning, ensuring better adherence to physical constraints and task requirements.

• Improving LLM Training with Ranking Rewards
  RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

  The paper introduces Ranking-based Reward Construction to bridge the gap between generative reward models and reinforcement learning algorithms.

• OneEmo: Unified Emotion AI Reasoning Model
  OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

  OneEmo is a 4.5B parameter multimodal model that improves emotion perception and understanding by using a novel reinforcement learning framework and a human-in-the-loop reasoning dataset.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• Improving Temporal Reasoning in Multimodal Models
  ChronoVision: Temporal Reasoning via Latent State Reconstruction

  ChronoVision introduces a visual-focused training framework to help multimodal large language models track and reason about continuous changes in images.

• Incentivizing LLM Reasoning via Pure Reinforcement Learning
  DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

  The paper introduces DeepSeek-R1, demonstrating that large language models can develop advanced reasoning capabilities through pure reinforcement learning without requiring prior supervised fine-tuning.

• Building Autonomous Agents for Minecraft
  Voyager: An Open-Ended Embodied Agent with Large Language Models

  Voyager is an embodied agent that uses GPT-4 to autonomously learn and store executable code skills for complex task completion in Minecraft.

• Constitutional AI for Harmless Assistants
  Constitutional AI: Harmlessness from AI Feedback

  The paper introduces Constitutional AI to train helpful and harmless AI assistants using self-critique and AI feedback instead of relying solely on manual human oversight.

• Teaching AI With Human Preferences
  Deep reinforcement learning from human preferences

  The paper introduces a method to train reinforcement learning systems on complex tasks using human feedback instead of well-specified reward functions.

• Avoiding Accidents in Machine Learning Systems
  Concrete Problems in AI Safety

  The paper identifies and analyzes five concrete problems related to accident risks in machine learning systems.

• End-to-End Training for Robotic Control
  End-to-End Training of Deep Visuomotor Policies

  The paper investigates whether training perception and control systems jointly end-to-end yields better performance than training each component separately for autonomous robotic operation.

• Mastering Go With Deep Neural Networks
  Mastering the game of Go with deep neural networks and tree search

  The paper introduces AlphaGo, a system that combines deep neural networks and tree search to master the game of Go.

• Interleaving Reasoning and Acting in LLMs
  ReAct: Synergizing Reasoning and Acting in Language Models

  The ReAct paradigm enables language models to solve complex tasks by interleaving verbal reasoning with external API actions to dynamically update plans.

• Playing Atari Games Using Deep Reinforcement Learning
  Playing Atari with Deep Reinforcement Learning

  The paper presents a deep learning model that successfully learns control policies directly from high-dimensional raw video data using reinforcement learning.

• Coach Models Improve Language Model Training
  LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

  The paper introduces an LLM-as-a-Coach method that replaces traditional scalar rewards in reinforcement learning with rich experiential knowledge to train open-ended task models.

• Improving LLM Training with Distilled Reinforcement Learning
  Distilled Reinforcement Learning for LLM Post-training

  The paper introduces a refined reinforcement learning method for LLM post-training that improves performance on math benchmarks by dynamically balancing teacher guidance and model autonomy.

• Structuring Robot Learning for Contact Tasks
  Data and Learning Where it Matters for Contact-Rich Manipulation

  The paper introduces a compositional framework that splits robot tasks into standard free-space planning and learned policies for contact-rich segments to improve reliability and generalization.

• Optimizing Agentic Reinforcement Learning with Muon
  When Does Muon Help Agentic Reinforcement Learning?

  The paper demonstrates that selectively applying the Muon optimizer to hidden weight matrices significantly boosts performance in agentic reinforcement learning tasks characterized by sparse rewards.

• Physics-Enhanced Reinforcement Learning for Real-Time Control
  Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

  The paper introduces Physics-EnhAnced Reinforcement Learning (PEARL), a new paradigm that addresses sample inefficiency and high dimensionality challenges in complex dynamical systems to enable real-time optimal control.

• Audio-Visual Flamingo for Complex Videos
  Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

  The paper introduces Audio-Visual Flamingo, an open model designed to improve joint perception, temporal alignment, and multi-event reasoning over long videos.

• Improving Agent Decision Making Through Self-Evolution
  SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

  The paper introduces a method called SEED that improves agentic performance by having the model analyze its own past trajectories to generate dense feedback for training.

• Improving Geo-localization by Reducing Landmark Bias
  HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning

  HoloGeo is a framework that improves image geo-localization accuracy by training models to reason beyond superficial visual landmarks using evidence-driven reinforcement learning.

• Learning Robot Tasks Using Reversed Data
  Reverse to Advance: Teleoperation-Cost Effective Hard Policy Learning from Reversed Easy Tasks

  The paper introduces a method to learn difficult robot manipulation tasks by automatically collecting and refining data from reversed easy tasks to lower teleoperation costs.

• Building Reliable Mobile Agents for Real World Tasks
  Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

  The paper introduces a framework for embodied agents that utilizes visual navigation, interactive intent disambiguation, and reinforcement learning to perform physical manipulation tasks in open world settings.

• Converting Document Images to Markdown Efficiently
  OvisOCR2 Technical Report

  OvisOCR2 is a model designed to parse visually rich documents into structured Markdown in a single pass.

• Stabilizing Inverted Pendulums With Lyapunov Rewards
  Lyapunov Exponent as Physics-Informed Dense Reward: RL Discovery of Stabilization Beyond the Kapitza Pendulum

  The paper uses the Lyapunov characteristic exponent as a reward signal to teach reinforcement learning agents how to stabilize an inverted pendulum with vertical motion.

• Automated Self-Improvement for LLM Judges
  RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges

  RecurSE enables LLM-based judges to improve their evaluation performance by creating a bounded, self-correcting feedback loop that eliminates the need for external gold standard rewards.

• Multi-tenant cloud service for VLA training
  JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models

  JoyNexus is a service-oriented framework that enables multi-tenant post-training for Vision-Language-Action models by decoupling compute resources from model execution.