You can efficiently erase stale or harmful information from an LLM's KV cache by learning to replace cached states rather than recomputing—enabling practical context correction in long-context applications without massive latency penalties.
KVEraser is a learned method for efficiently removing unwanted information from an LLM's cached key-value states after processing. Instead of recomputing all tokens after a deleted span (which is slow), it replaces only the cached states of the erased text with learned steering states, achieving near-full-recomputation quality with 3-4x speedup on long-context tasks.