Policy gradient
A family of reinforcement learning algorithms that directly optimize a model's output probabilities to maximize cumulative numerical reward signals.
What it is
Instead of predicting the next token or classifying an image directly, the model generates a sequence of actions, receives a scalar score representing quality at the end, and calculates gradients to make successful paths more likely. Because generation paths are long and stochastic, gradients have high variance, meaning millions of rollout steps are typically required to achieve stable convergence. Training often runs across hundreds of GPUs in parallel clusters, taking days or weeks to shift the underlying behavior of large models.
Why it matters
You care about policy gradients because this is the primary mechanism used to align base LLMs into helpful assistants through reinforcement learning from human feedback. If you are fine-tuning proprietary models or building custom behavior loops, ignoring this concept means you will misunderstand how safety guardrails, tone adjustments, and refusal behaviors are actually injected into the weights. Misconfiguring these training runs often leads to degraded reasoning capabilities or complete collapse of output quality.
In practice
You will rarely implement these gradient updates from scratch unless you work on infrastructure infrastructure for model training. Instead, you interact with this concept when setting hyperparameters like clip ranges and learning rates in libraries like TRL or DeepSpeed, or when monitoring training loss curves for sudden spikes in KL divergence.
The tradeoff
The main tradeoff is that these methods are notoriously unstable and sensitive to hyperparameters, often causing the model to learn shortcuts that maximize reward without actually solving the task.