Optimizing Mixed RL Rollout Throughput

Summary & Methodology Analysis

The paper addresses inefficiencies in mixed reinforcement learning (RL) rollout generation, where competing sessions suffer from KV-cache churn and throughput degradation due to unmanaged capacity competition. The proposed method, MISA-T, introduces adaptive session admission which sets caps for new sessions based on real-time demand and system pressure while ensuring existing sessions continue without interruption. This prevents the performance hits common in high-concurrency environments when prefix-aware routers fail to coordinate capacity sharing among diverse tasks.

To improve resource utilization, the system employs workload-aware capacity allocation that partitions protected KV-cache memory across specific classes such as RLVR, RLHF (a method for aligning model behavior with human preferences), and agentic workloads. MISA-T refines this by using residency-time-aware KV accounting, which weights demand by the observed session duration, explicitly incorporating the time spent on tool interactions. This ensures that capacity is reserved based on the actual temporal footprint of each workload class rather than just initial allocation requests.

Performance evaluations using Step3.7 show a 35.6 percent increase in rollout throughput and a 22.8 percent reduction in mean iteration time over 50-iteration experiments. Despite these gains, the system is subject to specific constraints. MISA-T relies on accurate workload labels and timely metrics provided by the serving layer. Because the method depends on these telemetry inputs, any processing delays at the serving layer can degrade the accuracy of the scheduling estimates, limiting the effectiveness of the capacity management logic.