Optimizing Vision Language Model Training Efficiency

Summary & Methodology Analysis

Rollplex addresses the inefficiency in reinforcement learning post-training for vision-language models, where traditional serial processing leaves GPU resources idle during rollout decoding. By decomposing the reference and training phases, the system moves prefix computation into the rollout decode window, effectively overlapping intensive prompt processing with decoding tasks. This approach ensures GPU capacity is utilized consistently, rather than forcing hardware to wait for sequential phase completion, without violating on-policy semantics. To manage the resulting memory pressure, the system uses a strategy that classifies intermediate states by producer-consumer lifetimes, keeping latency-critical boundary key-value states in high bandwidth memory (HBM) while offloading or recomputing larger training-specific states. The system leverages CUDA-VMM-backed memory allocation to change the physical residency of the working set dynamically, avoiding the overhead of rebuilding engine-level tensor objects. Additionally, it implements parallelism-aware weight sharing, which enables the use of the same physical actor storage for layout-compatible tensors across different tensor-parallel degrees, alongside a chunked optimizer update mechanism to stream 32-bit floating-point updates. While highly efficient, this architecture has a significant limitation in that it lacks independent recovery boundaries, meaning a failure in one colocated engine often causes the shared pool to fail entirely. The methodology was validated across a range of models including Qwen2.5-VL-32B, Qwen2.5-VL-7B, PaLM-E, RT-2, and OpenVLA, and evaluated against tasks like GSM8K, MATH, and LLaVA-Video-178K.