Efficient Reference Caching for Diffusion Transformers

Summary & Methodology Analysis

The paper introduces a method to enable efficient reference caching in diffusion transformers, which are neural networks that generate images by iteratively removing noise. By extending the input sequence with static text anchors, the model creates an isolated information pathway that allows reference image tokens to be processed once at the start of the denoising loop. These precomputed key and value states are then cached and reused throughout the remaining steps. Because these anchors are removed before the denoising stage, they do not add runtime overhead or increase the parameter count during inference. This solves the conflict where previous sparse attention methods, which optimize for memory by isolating tokens, caused a loss of instruction fidelity. The implementation involves modifying the attention structure, which creates a mismatch between inherited model weights and the new information flow. To correct this, the authors utilize a two-stage recovery process. First, they apply velocity distillation at data-derived states, followed by on-policy distillation using the model's own generated trajectories. This ensures the student model recovers performance despite the structural change. The authors evaluate their method on Qwen-Image-Edit-2511 and test generation quality across several benchmarks, including OmniContext, GEdit-Bench, and ImgEdit-Bench. There are specific limitations to the approach. The structural impossibility theorem, which provides the foundation for this method, assumes the attention mask is shared across all transformer layers and that at least two layers are used. Additionally, the initial change to the attention structure causes a drop in quality, meaning that distillation is a mandatory step rather than an optional optimization to reach the final benchmark scores.