Improving Diffusion Model Training with Uncertainty

Summary & Methodology Analysis

The paper addresses the instability in diffusion post-training caused by scalar reward models that lack uncertainty estimation. The core mechanism is SURE-LRM, which estimates both a mean reward and a sample-adaptive residual variance from noisy latents, prompts, and timesteps. By employing a heteroscedastic preference objective, the model accounts for the inherent noise in the generation process, which helps mitigate optimization errors and reward hacking during the training of generative models. This uncertainty-aware approach provides a more reliable signal for the reward process. Following the development of the reward model, the authors implemented SURE-REFL, a technique that freezes the SURE-LRM model to query multiple denoising transitions. By detaching the uncertainty estimates, the system generates reliability weights that normalize reward feedback. These weights are then used to modulate local gradient updates for the generator, ensuring that only high-quality, reliable feedback influences the model weights. The approach effectively replaces the standard, less stable, scalar feedback loop with one that dynamically scales updates based on model confidence. Despite these improvements, the current framework faces challenges regarding distribution shifts, which the authors identify as a primary area for future research. The paper does not specify the exact computational overhead or latency impact of calculating these uncertainty weights, nor does it provide details on the memory footprint required to maintain these transitions during training.