Not all timesteps and trajectories in diffusion model training contribute equally to learning—by selectively weighting informative steps and replaying valuable past samples, you can dramatically reduce the amount of human feedback needed to align diffusion models.
This paper improves the efficiency of reinforcement learning from human feedback (RLHF) applied to diffusion models by identifying that reward information is unevenly distributed across denoising timesteps and trajectories.