MemDecay: AI Agents Learn Which Memories Actually Matter
Key Takeaways
- ▸Different context types in LLM agents have dramatically different semantic half-lives—system prompts outlive scratch notes by 10x, a pattern MemDecay exploits
- ▸Region-aware KV cache eviction preserves semantic memory at full accuracy while baseline methods fail to retain critical information in more than half of test cases
- ▸The technique is training-free and calibration-based, making it practical to deploy across different model sizes and deployments without retraining
Summary
Researchers have introduced MemDecay, a novel training-free technique for managing key-value cache memory in large language model agents. The research addresses a critical bottleneck: as LLM agents accumulate diverse context—system instructions, plans, user interactions, retrieved documents, and tool outputs—their KV cache becomes prohibitively expensive, limiting deployment efficiency.
MemDecay uses semantic-aware eviction by assigning region-specific retention priorities to different types of context. The key insight: not all tokens need equal lifetimes. Testing on Qwen2.5 models revealed stark differences—system instruction tokens maintain relevance for 148-189 decoding steps, while temporary scratchpad tokens only need 14-16. By preserving critical semantic regions while aggressively pruning ephemeral content, MemDecay maintains full accuracy on essential information while reducing memory overhead. Notably, region-aware retention scales robustly as context windows grow, whereas simpler recency-based approaches collapse under larger contexts.
- Semantic structure—already available to the agent orchestrator—is a more robust signal than attention patterns alone for cache management
Editorial Opinion
MemDecay cuts through a real but overlooked problem: LLM agents treat all context equally when deciding what to forget. By respecting semantic structure—the fact that system prompts outlive notes—this work offers a principled path to far more efficient long-running agents. The distinction between what an LLM currently attends to and what it structurally needs is subtle but powerful; this paper suggests they're different retention signals worth managing separately.
