Stabilizing Inverted Pendulums With Lyapunov Rewards

Summary & Methodology Analysis

The paper tackles the control problem of stabilizing an inverted pendulum subject to vertical base motion. Instead of traditional reward engineering, the author utilizes the Lyapunov characteristic exponent (LCE), a scalar value that measures the rate of separation of infinitesimally close trajectories in a dynamical system, as a dense reward signal for the reinforcement learning (RL) agent. This approach focuses the agent on the stability properties of the system rather than relying on sparse positional feedback during training. By providing a continuous, dense signal, the agent is able to navigate the dynamics of the system more effectively during the learning process. The core outcome of this work is the emergence of the Kapitza pendulum oscillation, a known physical phenomenon where a pendulum remains upright through rapid vertical vibration of the base. The RL agent successfully discovered and exploited this mechanism to maintain balance in the upright position. As the extracted facts do not provide specific data on training duration, compute costs, model architecture details, or performance metrics like convergence rate, the relative efficiency of this approach compared to traditional RL reward functions remains unspecified by the paper. Furthermore, the paper does not list specific limitations or potential failure modes for the learned policy.