Safely Composing Multi Agent AI Policies

Summary & Methodology Analysis

The paper addresses a significant gap in multi-agent transfer learning: how to combine independent agent policies safely when teammate behavior varies. Traditional independent composition often breaks safety guarantees, while synchronized approaches lack the flexibility needed for heterogeneous tasks. To solve this, the authors implement MA-USFA (Multi-Agent Universal Successor Feature Approximators), which models successor features (a representation of long-term state occupancy and reward accumulation) while dynamically accounting for the task context of nearby teammates. This allows each agent to adapt its policy based on the specific behavior of those it interacts with.

The system utilizes a two-layered architecture. The lower layer consists of the MA-USFA models, and the upper layer is a learned composer. This composer selects library entries across agents and introduces cross-agent corrections, which are necessary because individual value models lack the capacity to capture inter-agent dependencies. During training, the composer is initialized using an independent transfer policy and trained with the value layer frozen, ensuring that the model achieves improvements on team objectives without the overhead of per-task adaptation.

Despite these advancements, the authors highlight inherent limitations in existing composition strategies. Synchronized composition is described as unconditionally safe but too rigid for heterogeneous objectives. Conversely, independent composition relies on strict alignment and value validity requirements that are not guaranteed in general multi-agent tasks. The paper demonstrates that relying on independent per-agent successor features when teammate behavior changes leads to staleness, which can cause performance to fall below that of basic library policies.