You can compress LLM KV caches by 3.9x without losing accuracy on long-context tasks by factorizing states into frequency components and adaptively assigning heads—no retraining needed.
FreqDepthKV compresses KV caches during LLM inference by splitting key-value states into shared low-frequency components and sparse residuals, then dynamically assigns attention heads to different compression modes based on their importance. This preserves accuracy on long-context tasks while reducing memory use by 3.9x and improving inference speed.