Accelerating Video Diffusion Transformers with Sparse Attention
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- LoSA achieves a 1.36x speedup on Wan2.1-1.3B with a minimal 0.06-point VBench Overall drop.
- Combining LoSA with feature caching techniques like D2Cache allows a 3.2x speedup on HunyuanVideo with a 0.02-point VBench Overall drop.
- The method remains training-free, avoiding the need for retraining or fine-tuning models.
- Performance is improved by freezing selective key and value blocks after an initial construction phase.
Summary & Methodology Analysis
Video diffusion models typically incur high latency due to the quadratic complexity of self-attention, an operation that calculates relationships between every token in a 3D sequence, over long sequences. LoSA addresses this by running dense self-attention for a few initial denoising steps to build an attention mask. At a construction step (t=3), the system calculates exact block attention masses for each layer, head, and query block. It then identifies the minimal set of key/value blocks required to satisfy a cumulative attention mass threshold, set by default to 0.99, and freezes these indices for all subsequent steps. In these later steps, the model only computes attention using the frozen retained keys while recomputing the query, key, and value vectors from the current hidden states.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem LoSA solves?
It solves high sampling costs in video diffusion transformers caused by the quadratic complexity of self-attention over long 3D sequences.
Q2. Is this a training-based approach?
No, LoSA is a training-free method for accelerating video diffusion inference.
Q3. Does LoSA work with other acceleration methods?
Yes, it can be combined with standard feature caching techniques like D2Cache.
Q4. How does LoSA select which attention blocks to keep?
It measures the exact block attention mass at a construction step (t=3) and retains the smallest set of blocks that meet a cumulative mass threshold of 0.99.
Q5. What is the computational cost of the initialization phase?
The construction step adds a one-time computational overhead equivalent to one denoising step.
Q6. Are the selected block indices updated throughout the generation?
No, the selection is fixed after the initial construction step.
Q7. What performance results were observed on HunyuanVideo?
When combined with feature caching, it reaches a 3.2x speedup with a 0.02-point VBench Overall drop.
Q8. Which specific models were tested?
Testing was performed on Wan2.1-1.3B, Wan2.1-14B, HunyuanVideo-13B, and CogVideoX.
Q9. Does the paper define the memory savings achieved?
The paper does not specify the exact memory savings.