You can make models significantly more compressible during training with a simple regularizer that costs less than 1% extra compute and doesn't require changing your model architecture or doing expensive matrix decompositions.
SLORR is a training-time regularization method that makes neural networks easier to compress using low-rank factorization. Unlike existing approaches, it works directly on weight matrices without requiring expensive computations, architectural changes, or cached data, adding minimal training overhead while improving how well compressed models perform.