Policy

What it is

In practice, a policy is a function or model that accepts an input state and returns a probability distribution over possible actions. If you are using a transformer for agentic workflows, the policy is effectively the weights of the model that decide the next token to emit. These models are often trained via reinforcement learning where the policy is iteratively updated to maximize a cumulative numerical reward. The size of the policy is equivalent to the parameter count of the underlying model, ranging from millions to hundreds of billions of parameters.

Why it matters

Understanding policy is critical when you deploy agents that interact with external APIs or users because the policy determines the reliability and safety of those interactions. If you do not monitor the policy behavior, an agent may fall into action loops or take unauthorized actions. You should care because tuning the policy, often through fine-tuning or prompt engineering, is the primary way to reduce error rates in autonomous workflows. Ignoring the policy risks unexpected runtime behaviors that can lead to high latency or excessive API costs due to suboptimal decision-making.

In practice

In production, you influence the policy by updating the system prompt, providing specific in-context examples, or swapping to a fine-tuned model version. You observe policy behavior by tracking action success rates or using logging to audit the agent's decision paths against your intended logic. When the policy fails, you generally adjust the training data or the system instructions rather than modifying the inference engine directly.

The tradeoff

The primary tradeoff is between a policy that is too rigid, causing failure on edge cases, and one that is too flexible, resulting in unpredictable or hallucinated actions.

Where it appears

Research summaries that use Policy, each linked to its source paper.

• Jointly Training AI to Create and Use Tools
  Joint Optimization of Tool Creation and Use for Large Language Model Agents

  The SMITH framework uses reinforcement learning to train a single language model to both synthesize reusable Python tools and apply them effectively to solve complex tasks.

• Making Static AI Training Environments Dynamic
  EnvHarness: Awakening Static Worlds for Agent Learning

  EnvHarness and EnvRigger dynamically modify static environments to improve LLM agent training efficiency and performance.

• Improving Skill Selection for AI Agents
  SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

  SkillGate optimizes agent performance by partitioning training signals to stop task outcomes from interfering with how agents select procedural skills.

• Improving Multi-Reward Language Model Training
  Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

  The paper introduces SA-MRPO, a method that dynamically reweights reward objectives during reinforcement learning to prioritize underperforming metrics instead of treating all goals as equally weighted.

• Safely Composing Multi Agent AI Policies
  Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning

  The paper introduces a method for composing reusable AI policies in multi-agent environments that maintains safety and flexibility without requiring per-task retraining.

• Robotic Foundation Model for Goal Navigation
  RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

  RynnValue is an open-source foundation model that improves robotic task success rates by predicting the remaining distance to a goal using visual observations.

• Simpler Control of Language Models
  Direct Preference Optimization: Your Language Model is Secretly a Reward Model

  Researchers developed a simpler method to control language model behavior directly from human preferences without training a separate reward model or using reinforcement learning loops.

• Teaching AI With Human Preferences
  Deep reinforcement learning from human preferences

  The paper introduces a method to train reinforcement learning systems on complex tasks using human feedback instead of well-specified reward functions.

• Coach Models Improve Language Model Training
  LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

  The paper introduces an LLM-as-a-Coach method that replaces traditional scalar rewards in reinforcement learning with rich experiential knowledge to train open-ended task models.

• Physics-Enhanced Reinforcement Learning for Real-Time Control
  Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

  The paper introduces Physics-EnhAnced Reinforcement Learning (PEARL), a new paradigm that addresses sample inefficiency and high dimensionality challenges in complex dynamical systems to enable real-time optimal control.

• Optimizing Language Model Reasoning During Inference
  TTPO: Test-Time Policy Optimization

  TTPO improves the reasoning accuracy of language models during test time by using label-free policy optimization that bypasses the need for manual ground-truth data.

• Improving Agent Efficiency with Stream Aligned Optimization
  SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

  SPO++ is a refined policy optimization framework that increases online learning efficiency for language agents by aligning data tracking with event timing.

• Optimizing Reinforcement Learning for Faster Training
  WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

  WarpSAC is a scalable reinforcement learning framework that adapts its architecture based on available compute resources to accelerate training and improve deployment success.

• Improving Multi-Turn AI Agent Credit Assignment
  IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

  The paper introduces IAPO, a method that improves agent training by redistributing reward credit based on how agent actions influence one another within multi-turn service workflows.

• Improving AI Reasoning Through Verifiable Distillation
  On-policy Distillation with Verifiable Reward

  The paper introduces a method called OPDVR that aligns reinforcement learning signal with task success during model distillation to improve reasoning performance.

• Securing AI Agents Using On-Policy Distillation
  SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

  SecOPD improves AI agent security against adaptive prompt injection by using on-policy distillation to provide fine-grained training signals that distinguish between trusted instructions and malicious data.

• Improving Robot Policies Without Retraining
  Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

  The paper presents a method that enables robot policies to self-improve through iterative deployment without the need to modify the original policy weights.

• Evaluating Cache Eviction Policies for LLMs
  Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

  The paper systematically evaluates various eviction policies for semantic LLM caching and finds that the standard Least Frequently Used approach remains highly effective compared to more complex alternatives.

• Optimizing Autoregressive Robotic Policy Deployment
  RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

  RoMAN-Flow introduces post-training optimization and distillation techniques to eliminate the sequential sampling latency inherent in autoregressive normalizing flows for robotic control.

• Enforcing Organizational Policies in LLM Agents
  PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

  PolicyGuide introduces a workflow-based verification system that uses an external runtime graph to enforce organizational compliance in LLM agents.

• Safely Shaping Agent Rewards with LLMs
  Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

  The paper demonstrates that using LLM progress scores to shape rewards for reinforcement learning agents does not alter the underlying optimal policy of the agent.

• Transferring Reasoning Capabilities Between Language Models
  SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

  The paper introduces a method called SimpleOPD that distills advanced mathematical reasoning from a high-performance teacher model into smaller student models across different architectures.

• Improving AI Vision Without External Data
  Self-Supervised Visual On-Policy Distillation

  The paper introduces a self-supervised method to improve vision model performance by distilling knowledge from a student to an EMA teacher without needing ground-truth labels or extra rewards.

• Improving Model Distillation via Task Sensitivity
  CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

  The paper introduces a method called CROP to selectively focus model distillation on task-relevant information by measuring sensitivity to counterfactual prompts.

• Teaching Robots Dexterous Manipulation with Teleoperation
  NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation

  NestDex improves dexterous robot manipulation by combining pre-trained hand skill policies with operator teleoperation using an action-compressing variational autoencoder.

• Learning Visual Bottlenecks for Robot Policies
  Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning

  The paper introduces a Seeker module that learns to focus robot vision on relevant spatial regions, significantly increasing success rates in complex environments.

• Making AI Models Learn Their Own Context
  Latent On-Policy Self-Distillation

  The researchers developed a method to replace hand-designed improvement rules with a system that learns to generate its own contextual guidance for model training.

• Self-Evolving GUI Visual Grounding Agents
  Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

  The paper introduces a framework that allows GUI visual grounding models to continuously improve after deployment by learning from their own exploration failures through reflection-guided self-distillation.

• Improving Robot Policy Robustness with World Models
  JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

  The paper introduces JEPA-WAM, a framework that integrates spatially structured world modeling with vision-language-action policies to improve performance and robustness against distribution shifts.

• Measuring Cross-Lingual Consistency in AI Agents
  Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

  The paper introduces a new measurement protocol to evaluate if AI agents execute the same tool-use action sequences across different languages.

• Standardizing Robot Policy Evaluation and Deployment
  XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

  XPolicyLab introduces a unified standard and client/server architecture to resolve the fragmentation in robot policy integration and evaluation.

• Fixing On-Policy Distillation Mismatch
  Mismatch Matters: On-Policy Distillation Beyond Token Agreement

  The paper introduces TIDE, a method to fix model distillation failures caused by degenerate token agreement and teacher-student mismatch.

• Adaptive Supervision for Better Reasoning Models
  DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

  The researchers developed a method called DASH that dynamically adjusts how reasoning models learn from their own outputs to produce more accurate results.

• Improving Multilingual Reasoning in Language Models
  RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

  The paper introduces RP-OPSD, a method that improves how language models transfer English reasoning skills to low-resource languages by selectively applying privileged distillation based on reasoning-pivot signals.

• Improving Multimodal Models Using Visual Distillation
  OPD-V: Visual On-Policy Self-Distillation with Modality Balance

  OPD-V improves multimodal model performance and reduces latency by using visual-based self-distillation to balance how the model uses image and text data.

• Better Update Methods for Reinforcement Learning
  Proximal Policy Optimization Algorithms

  The authors propose a novel surrogate objective function that enables multiple epochs of minibatch updates for proximal policy optimization, improving efficiency and stability.

• Optimizing Cooperative Multi-Agent Reinforcement Learning
  Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

  The paper provides a design rule for aggregating agent data that prevents unstable learning in large cooperative multi-agent systems.

• Using RAG for Better Action Selection
  Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inference

  The paper introduces a causal inference framework to improve how RAG systems select actions by treating vector search as a nearest neighbor matching process.

• Efficient Robotic Control via Patch Representations
  Patch Policy: Efficient Embodied Control via Dense Visual Representations

  The paper introduces a method that replaces compressed global visual vectors with uncompressed patch tokens to improve robotic manipulation precision while maintaining high computational efficiency.

• Identifying Optimal Policies with Fewer Samples
  Non-Asymptotic Best Policy Identification Guarantees in Online Reinforcement Learning

  The paper introduces the first non-asymptotic analysis for the Navigate and Stop algorithm to identify optimal policies in online reinforcement learning with rigorous confidence bounds.

• Improving Model Reasoning With On Policy Delta Distillation
  On-Policy Delta Distillation

  The paper introduces On Policy Delta Distillation, a new method that improves how reasoning capabilities are transferred from a teacher model to a student model.

• Improving Agent Decision Making Through Self-Evolution
  SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

  The paper introduces a method called SEED that improves agentic performance by having the model analyze its own past trajectories to generate dense feedback for training.

• Learning Robot Tasks Using Reversed Data
  Reverse to Advance: Teleoperation-Cost Effective Hard Policy Learning from Reversed Easy Tasks

  The paper introduces a method to learn difficult robot manipulation tasks by automatically collecting and refining data from reversed easy tasks to lower teleoperation costs.

• Recovering Pruned LLM Generation Capabilities
  ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  ShortOPD uses a dynamic distillation strategy to fix structural collapse in pruned LLMs by adjusting training rollouts based on model output quality.

• Accelerating Robotic Action Models for Real-time Control
  GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch

  GigaWorld-Policy-0.5 improves real-time robotic control by decoupling action generation from future video simulation using a specialized Mixture-of-Transformers architecture.

• Improving Model Training with Trust Regions
  Trust-Region Behavior Blending for On-Policy Distillation

  The paper introduces a method called Trust Region Behavior Blending to stabilize on-policy distillation by controlling how teachers supervise student models during early training.