Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Summary

This paper explains why neural networks need to organize information in a specific geometric way to recognize familiar concepts in new combinations. The researchers prove that for a model to generalize to unseen combinations of concepts, its internal representations must decompose into separate, perpendicular components for each concept.