Fixing Visual Artifacts in Latent Diffusion

Summary & Methodology Analysis

Score Distillation Sampling (SDS) uses a pretrained diffusion model, a statistical engine that generates data by iteratively refining noise, to optimize images. A core issue is that SDS often operates within the latent space, which is a compressed representation of data. Because the VAE (a neural network used to map between image pixels and compressed latent space) is underconstrained, updates in latent space can result in pixel drift, creating artifacts because the encoder does not strictly supervise the resulting pixel color and texture values. PixSDS solves this by calculating the update direction in latent space, decoding it through the VAE to generate a look ahead image, and then defining a clean direction in pixel space based on the difference between the current image and this look ahead image. The final update is a combination of the original gradient and this normalized VAE consistent signal. This process ensures that the optimization path remains visually coherent. PixSDS performs well in 2D benchmarks, hitting an FID score of 223.021 with SGD versus 431.877 for standard SDS. It also produces cleaner results in 3D generation pipelines like DreamGaussian and LucidDreamer. However, it is not a perfect replacement for direct sampling from a pretrained diffusion model, which remains the higher quality standard for image generation. Furthermore, the paper notes that the method may struggle to preserve specific visual attributes for certain prompts, such as adding a white hair attribute to an Ironman character figure. The paper does not specify the exact computational overhead or latency impact of the extra decoding step required by this method.