Linear attention mechanisms can match standard transformer performance if you design the state updates correctly; the paper shows which architectural choices matter most for maintaining accuracy while cutting inference cost.
This paper analyzes why transformer self-attention is expensive and proposes a linearized alternative that reduces computational cost from quadratic to linear. By studying how attention mechanisms work mathematically, the authors identify key design principles—like using delta-style updates and sink tokens—that preserve model quality while dramatically speeding up inference on long documents.