Token-adaptive KV cache compression with cross-layer factorization can cut memory use by 8x while maintaining retrieval accuracy—enabling faster long-context inference without model retraining.
DepthWeave-KV compresses the key-value caches that slow down long-context language model inference by sharing low-rank representations across transformer layers while keeping token-specific details where they matter most. It uses a smart router to allocate more storage to important tokens and adapts compression on-the-fly during generation, achieving 8.3x memory reduction without retraining.