Optimizing Video Diffusion Model Inference Latency

Summary & Methodology Analysis

The researchers identified a systems bottleneck in Video Diffusion Transformers where Top-p sparse attention (a mechanism that selectively computes only the most relevant relationships between tokens to save compute) creates uneven workloads across heads. This imbalance results in rank-level stragglers during distributed execution across multiple GPUs. To solve this, the authors implemented an adaptive sparse-routing frontend that uses Top-p routing, a Top-k safety floor, and Hilbert curve-based block organization to construct sparse masks and metadata efficiently. This setup prepares the workload by defining importance rankings before computation begins. The framework then employs Runtime Load Balancing to profile the actual work per head and migrates overloaded tasks via peer-to-peer communication to minimize the critical path. Additionally, the system includes Slack-Aware Sparse Augmentation, which reclaims idle capacity on non-critical ranks to process extra high-value sparse blocks, further optimizing resource utilization. The paper utilizes an Overlap mechanism to hide the costs of scheduling, communication, and augmentation by performing these tasks concurrently with primary computations via CPU-GPU and computation-communication overlaps. While effective, the framework is limited by its reliance on sequence length and sparsity patterns, showing decreased effectiveness for short sequences or workloads with weak attention sparsity. The paper does not specify the exact hardware configurations used for these measurements beyond referencing multi-GPU settings.