Improving Video Generation Speed and Diversity

Summary & Methodology Analysis

Video generation often forces a choice between diversity, where a model generates a broad range of creative outputs, and fidelity, where outputs are visually sharp but potentially repetitive. The DUET method solves this by splitting the generation task into two stages. It assigns the high-noise interval from 1 to τ to a diversity-preserving distillation expert (sCM), which produces an initial prediction. This prediction is then re-noised to the switch time τ and passed to a fidelity-oriented distillation expert (DMD) for final refinement. This relay sampling approach allows the system to leverage the strengths of both paradigms without the optimization instability often found when combining loss functions directly. The architecture utilizes Reinforcement Learning (RL) to guide expert adaptation, applying a reward-weighted objective to the sCM expert to ensure it maintains structural diversity while the DMD expert adapts its refinement process to the relayed outputs. The training loop continues by optimizing the DMD expert with its distribution-matching objective, ensuring it is fine-tuned to the specific inference distribution produced by the sCM expert. Evaluations on the Wan2.1-T2V-1.3B model demonstrate significant improvements in both diversity and visual quality metrics. Current implementation is limited to the Wan2.1-T2V-1.3B model, and the paper does not specify if this technique is compatible with other model architectures.