Linear attention in graph transformers can only learn averaged denoising filters, but Graph Convolutional Attention leverages spectral information to adapt denoising to each graph's unique structure—improving both performance and inference speed.
This paper explains why standard attention mechanisms struggle with graph denoising and proposes Graph Convolutional Attention (GCA), which uses the graph's spectral properties to denoise more effectively. GCA provably outperforms linear attention and works well in graph diffusion models, offering both theoretical guarantees and practical speedups.