You can reuse existing discriminative models (classifiers) for generative tasks by freezing them and training lightweight adapters, cutting the model footprint in half while keeping performance—useful when you already have trained classifiers lying around.
This paper shows how to repurpose a pre-trained speech classifier for generating speech by attaching a lightweight denoising network on top of it. Instead of training separate classifier and diffusion models, the authors freeze the classifier and train only a small adapter to guide generation, reducing memory and computation while maintaining high speech quality.