Physics-Enhanced Reinforcement Learning for Real-Time Control

Summary & Methodology Analysis

Traditional reinforcement learning, a machine learning technique where an agent learns to make decisions by trial and error to maximize a reward, struggles with sample inefficiency and high dimensionality when applied to nonlinear, complex, high-dimensional, and parametric dynamical systems. To solve this, the paper presents Physics-EnhAnced Reinforcement Learning (PEARL), a paradigm designed for real-time optimal and adaptive closed-loop control. The existing mechanics involve an agent taking actions using policy parameters while the environment evolves according to transition dynamics, evaluating task-specific loss or reward functions, computing policy gradients over short horizons using automatic differentiation through the dynamics to prevent gradient instabilities, and updating an online critic or value network by minimizing mean squared error with temporal-difference learning targets.

Building on these foundations, PEARL introduces novel contributions, including updating policy parameters online after every short horizon towards a gradient descent direction using an actor-adjoint method. It approximates the terminal adjoint of each short horizon using a neural network called the adjoint network. Furthermore, the adjoint network is trained by generating target adjoint variables through a temporal-difference scheme based upon the adjoint equation. The paper evaluates PEARL against several models and datasets, specifically PPO, TD3, BPTT, SHAC, PEARL, and SHAC-MOD, demonstrating its effectiveness across different reward structures.

Regarding performance and limitations, the paper notes that PEARL outperforms model-free algorithms like PPO and TD3, as well as automatic differentiation alternatives including BPTT, truncated BPTT, and SHAC, in terms of training and evaluation rewards on the leader-follower game with dense and sparse rewards. In the high-dimensional mean-field leader-follower game with dense rewards, PEARL achieves the best-performing rewards during evaluation. However, the study identifies a notable limitation: model-free reinforcement learning algorithms suffer from severe sample inefficiency and are limited to sparse sensors and actuators due to the curse of dimensionality. The paper does not specify precise hardware requirements, dollar costs, or exact wall-clock training times.