Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems
Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance
Read on arXivLatest papers
- FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI SystemsAI often names the right cause of a financial mismatch without ever finding the proof for it
- Looped Language Models Improve Compositional Tool CallingAI models that rethink their own answers multiple times get better at chaining tools together
- FACET: Preserving Source Intent and Executable State in Terminal Task SynthesisFACET builds internally consistent terminal-task 'exam sets' to train command-line AI agents
- FM-Bench: A Benchmark for Long-Horizon Management with Competing AgentsHanding an AI a football club to run for 20 years reveals that winning comes from management habits, not raw model power
- Bridging Search and CRM: Productionizing AI Product Research Agents for Customer Re-EngagementAI agents win back window-shopping customers by chasing them down on WhatsApp
- Adversarial Review: Structured Disagreement for Grounded Agentic Code ReviewFor AI code review, one reviewer plus one critic beats piling on more agents
- GenEx: A Graph-Based Representational Paradigm for SARS-CoV-2 Variant Detection via Codon Co-occurrence NetworksTurning viral gene sequences into codon relationship maps to tell coronavirus variants apart
- Alignment Is All You Need: Instruction-Free Training for General Audio-Language ModelsA frozen language model plus one lightweight connector is enough to build a capable audio-understanding AI
Latest from METAL LAB
- NVIDIA's 300 Verified Skills Lift Correctness by 41 Points
- Wave your hand at a webcam, hear a theremin: browser instrument released
- Meta AI launches desktop app for Mac, can read an entire app window
- Factory Commits $100M to Partner Network, Pushes to Scale Software Factories
- SpaceX approached Cognition for acquisition four days after closing Cursor deal