One Layer Is Enough: Adapting Highly Trained Visual Encoders for Image Production
Generative generative models (eg, distribution models) typically operate on compressed latent spaces to measure training efficiency and sample quality. Correspondingly, there has been a growing interest in using high-quality pre-trained visual representations—either by aligning them within VAEs or directly within the generative model. However, correcting such representations is still a challenge due to the important mismatch between the features targeted for understanding and the latent spaces suitable for generation. Representational encoders benefit from high-dimensional masks that capture various views of masked areas, while production models prefer low-dimensional masks that should faithfully preserve the injected sound. This disparity has led to previous work relying on complex objectives and structures. In this work, we propose FAE (Automatic Feature-Encoder), a simple yet effective framework that adapts pre-trained visual representations into low-dimensional hidden objects suitable for production using as little as one layer of attention, while preserving sufficient information for both reconstruction and understanding. The key is to combine two different depth decoders: one is trained to reconstruct the original feature space, and the second takes the reconstructed features as input for image generation. FAE is general—it can be reinforced with self-supervised embeddings (eg, DINO, SigLIP) and it is connected to two different productive families–distribution and general flow models. In all conditional and text-to-image benchmarks, FAE achieves strong performance. For example, for ImageNet 256×256, our diffusion model and CFG obtain near-modern FID now of 1.29 (800 epochs) and 1.70 (80 epochs). Without CFG, FAE achieves a modern FID of 1.48 (800 epochs) and 2.08 (80 epochs), showing both high quality and fast learning.
