Modality interference—caused by gradient conflicts between audio and semantic processing—is the root cause of poor full-duplex SLM performance; hierarchical parameter separation solves this while maintaining cross-modality coherence.
This paper identifies and solves a critical problem in full-duplex spoken language models: when audio and text processing share the same neural network layers, they create conflicting gradients that degrade performance.