Distilling Knowledge Between Different AI Models

Summary & Methodology Analysis

Any-OPD solves the technical bottleneck in distillation where student and teacher models must share identical underlying configurations. By introducing a heterogeneous distillation framework, the method allows for transferring performance from complex models like FLUX.1-dev to lighter variants like SD3.5-Medium. This is achieved through a multi-stage process that avoids direct architecture dependency, making it feasible to distill across different model families that would otherwise be incompatible due to mismatched noise schedules or latent space structures. The methodology begins with an offline anchoring stage to map the teacher's output distribution into the student's latent space. In the subsequent on-policy stage, the student generates a sample, which is projected onto the teacher's image manifold using a noise-and-denoise map specific to the teacher. Instead of relying on rigid solver index alignment, the process routes corrections through the student transformer using noise-level matching to maintain structural consistency. Finally, the system computes a loss based on the cosine distance between the student output and the teacher projection within a frozen, model-agnostic space provided by DINOv2 CLS embeddings. This ensures that the student learns to approximate the teacher's behavior without requiring matching internal architectures. The quality of this distillation is primarily limited by the denoising budget available to the teacher model, which determines the ceiling for the improvements in metrics like HPSv3 and PickScore.