RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction

arXiv:2608.012472026-08-01

Adding 8 learned 'restore tokens' brings heavily compressed KV caches back close to full-cache quality

Long-context LLMs shrink their KV cache (stored key-value computations from past tokens) to save memory, but aggressive compression can badly hurt answer quality. RestoreKV keeps existing selection methods for which KV pairs to keep, and additionally runs a single pass where a handful of learned tokens attend to the full cache before eviction to generate a compact 'restore cache' that fills part of the same fixed budget. On Qwen3-4B, this raised KVzip's RULER-4K accuracy from 38.2 to 73.2 at a tight budget, adding less than 0.5% extra time to the compression process.

METAL LAB explanatory visual

How RestoreKV builds its budget-matched cache

Evidence statusMeasured results reported

1. Context prefillThe model processes the long input once to produce the full KV cache of stored key-value pairs.
2. Restore pass8 trained restore tokens, with LoRA temporarily enabled, attend to the entire full KV cache to generate a compact restore cache before eviction.
3. Base eviction unchangedThe existing scorer (e.g., KVzip) picks which original KV pairs to retain, leaving room for the restore cache within the same total budget.
4. Budget-matched combinationRetained original KV pairs and the generated restore cache are combined into one final cache of the same total size as before.
5. Query and decodingLoRA is turned off; the frozen original model answers arbitrary future queries using the combined cache.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

Query-agnostic KV eviction compresses a context once for reuse across future queries, but performance collapses under aggressive compression budgets.
RestoreKV keeps existing selection-based methods (like KVzip) unchanged for choosing which original KV pairs to keep, and adds a LoRA-adapted single pass where 8 learnable restore tokens attend to the full KV cache to generate a compact, context-conditioned 'restore cache'.
This restore cache occupies part of the same fixed KV budget (it replaces some retained slots), so total memory cost stays the same; LoRA is active only during this one-time restore pass, while all later query processing and decoding use the original frozen model.
Training uses self-distillation: only the restore-token embeddings and LoRA adapters (0.4% of parameters) are optimized so the compressed-cache model's answer distribution matches a frozen full-cache teacher's distribution.
Tested across four backbones (Qwen3-0.6B/4B/8B, Llama-3.1-8B) and four benchmarks (RULER-4K, QASPER, QuALITY, LongHealth) with five base eviction methods (KVzip, KVzip+, ContrastKV, SnapKV, H2O), improving nearly all paired settings.

Figure 1: RestoreKV narrows the gap to full-cache behavior under aggressive compression at negligible one-time cost. Applied to KVzip+, RestoreKV reaches 86.4 RULER accuracy at 16× compression on the KVPress Benchmark while adding negligible compression-time overhead.

Table 1: Generalization across KV cache eviction methods. RestoreKV improves five different base methods on Qwen3-4B under matched KV budgets, with larger gains typically observed at tighter budgets. Colored subscripts indicate absolute changes from the corresponding baseline.
Method	RULER-4K	QASPER	QuALITY	LongHealth
r=0.2	r=0.1	r=0.05	r=0.2	r=0.1	r=0.05	r=0.2	r=0.1	r=0.05	r=0.2	r=0.1	r=0.05
KVzip	91.4	80.1	38.2	38.2	33.9	18.0	66.0	58.5	46.3	71.5	70.0	61.0
KVzip + Ours	93.5 +2.1	88.8 +8.7	73.2 +35.0	39.0 +0.8	37.7 +3.8	28.7 +10.7	67.0 +1.0	63.3 +4.8	52.4 +6.1	75.8 +4.3	73.8 +3.8	67.5 +6.5
KVzip+	91.3	84.8	51.6	38.1	32.4	23.3	66.5	60.0	51.9	73.2	67.8	63.2
KVzip+ + Ours	93.3 +2.0	89.7 +4.9	70.7 +19.1	40.9 +2.8	38.5 +6.1	31.6 +8.3	66.8 +0.3	62.9 +2.9	57.9 +6.0	74.2 +1.0	73.0 +5.2	71.5 +8.3
ContrastKV	91.6	80.7	38.0	40.1	31.9	17.8	66.2	62.2	50.7	74.2	69.0	49.0
ContrastKV + Ours	92.3 +0.7	84.3 +3.6	40.2 +2.2	40.9 +0.8	36.0 +4.1	21.6 +3.8	67.6 +1.4	63.3 +1.1	51.9 +1.2	75.2 +1.0	70.5 +1.5	54.0 +5.0
SnapKV	33.8	20.6	12.7	22.4	19.0	14.3	53.9	53.0	42.5	41.0	36.8	36.2
SnapKV + Ours	37.7 +3.9	26.3 +5.7	14.3 +1.6	29.9 +7.5	24.0 +5.0	20.7 +6.4	59.4 +5.5	51.8 −1.2	49.9 +7.4	51.8 +10.8	41.8 +5.0	38.5 +2.3
H2O	8.0	3.5	3.2	22.0	13.8	14.4	59.2	50.4	41.5	60.2	48.8	37.0
H2O + Ours	17.3 +9.3	11.6 +8.1	5.7 +2.5	30.4 +8.4	24.7 +10.9	21.3 +6.9	60.9 +1.7	54.9 +4.5	47.7 +6.2	64.8 +4.6	59.0 +10.2	46.8 +9.8

Figure 2: Overview of RestoreKV. After context prefill, a single LoRA-adapted restore pass processes n restore tokens (n=8 by default) with causal access to the full KV cache, generating nLH context-conditioned KV pairs. These restore states are combined with the retained context states under the same total KV budget. LoRA is active only during restore-cache generation; subsequent query processing and decoding use the original frozen backbone. Training (offline): The restore-token embeddings and LoRA adapters (0.4% of the 4B backbone) are optimized by self-distillation from the frozen full-cache teacher, Inference: The trained restore tokens and LoRA adapters generate the restore cache once for each new context.

Table 2: Component and LoRA-target ablations on RULER-4K (Qwen3-4B). The first row is KVzip without a restore cache; the last row is the full RestoreKV configuration. “Fixed” replaces all eight learned restore-token embeddings with the embedding of the line-break token \n.
			KV budget ratio r
Restore emb.	LoRA target	#Params	0.2	0.1	0.05
–	–	–	91.4	80.1	38.2
Learned	–	0.02M (0.0005%)	92.7	82.6	42.1
Learned	q/k/v	4.0M (0.1%)	93.4	88.2	72.4
Learned	q/k/v/o	5.9M (0.15%)	93.5	88.4	72.0
Fixed	q/k/v/o+MLP	16.5M (0.4%)	93.5	88.0	71.9
Learned	q/k/v/o+MLP	16.5M (0.4%)	93.5	88.8	73.2

Figure 3: Performance across four model backbones and four benchmarks. RestoreKV and RestoreKV+ are applied to KVzip and KVzip+, respectively, under the same KV budget. Both methods reduce compression-induced performance loss, with larger gains under tighter cache budgets.

Table 3: Effect of full-context conditioning on RULER-4K with Qwen3-4B. Generating the restore cache from the full cache before eviction adds 8.8 points at r=0.05 over generating it from the already evicted cache.
	KV budget ratio r
Method	0.2	0.1	0.05
KVzip (n=0)	91.4	80.1	38.2
Generated from evicted cache	93.7	87.6	64.4
Generated from full KV cache	93.5	88.8	73.2

Figure 4: Comparison with a Synthesized-Cache Baseline on Qwen3-4B. RestoreKV+ consistently improves token-eviction baselines and remains competitive with AM-fast, a strong synthesized-cache method, at ∼13× lower compression time on 4K-token contexts.

Table 4: Ablation on the number of restore tokens (RULER-4K, Qwen3-4B). All variants share the same training recipe and total KV budget.
	KV budget ratio r
Method	0.2	0.1	0.05
n=0 (KVzip)	91.4	80.1	38.2
n=1	93.0	85.3	65.3
n=2	93.4	87.6	70.4
n=4	93.4	87.0	68.5
n=8 (default)	93.5	88.8	73.2
n=16	93.3	87.8	69.1

Figure 5: Layer-wise context-attention recovery on RULER-4K (Qwen3-4B, r=0.05). KVzip substantially suppresses attention from the final query token to non-sink context KV pairs. RestoreKV recovers part of the lost attention (shaded), increasing the average context-attention mass from 0.56% to 1.30% and reducing KL divergence from the full-cache model by 16% (→0.167).

Table A: Ablation on the training ratio range (RULER-4K, Qwen3-4B). RestoreKV is trained by sampling the retention ratio from 𝒰⁡(rmin,rmax). The first row reports the KVzip baseline without restoration.
Training ratio range	KV budget ratio r
rmin	rmax	0.4	0.2	0.1	0.05
-	-	93.5	91.4	80.1	38.2
0.025	0.25	94.3	93.5	88.8	73.2
0.025	0.50	94.1	93.1	86.1	63.5
0.050	0.25	94.3	93.2	86.5	65.6
0.050	0.50	94.0	92.8	84.5	51.5

Figure 6: Efficiency analysis (Llama-3.1-8B-Instruct, 32K context). (a) Restore-cache generation adds only ∼0.04 s (0.5%) and 84 MB (0.4%) on top of any base scorer. (b) Attention latency and KV memory are nearly identical to the base method.

Table B: Effect of the query-position offset (RULER-4K, Qwen3-4B). Offset-only KVzip applies the same +8 query offset as RestoreKV without adding restore states. Its performance remains close to standard KVzip, suggesting that the positional offset alone is unlikely to account for RestoreKV’s improvement.
Variant	Restore states	Query offset	KV budget ratio r
0.2	0.1	0.05
KVzip	No	0	91.4	80.1	38.2
Offset-only KVzip	No	+8	90.5	80.8	36.9
RestoreKV	Yes	+8	93.5	88.8	73.2

Figure A: RestoreKV composes with Fast KVzip, a learning-based scorer (RULER-4K, Qwen3-8B). At 16×, Fast KVzip drops to 46.7, RestoreKV restores it to 79.3, above plain KVzip (59.0).

Table C: Effect of training–inference evictor mismatch (RULER-4K, Qwen3-4B). The inference evictor is fixed to KVzip, while RestoreKV is trained using the evictor in the first column.
Training evictor	Inference evictor	KV budget ratio r
0.20	0.10	0.05
-	KVzip	91.4	80.1	38.2
SnapKV	KVzip	93.5	81.6	42.0
KVzip	KVzip	93.5	88.8	73.2

Figure B: SCBench results on long-context tasks (Qwen3-4B).

Table D: LongBench results over 16 tasks under aggressive KV compression. Categories are single-document QA (S-QA), multi-document QA (M-QA), summarization (Summ), few-shot learning (Fewshot), synthetic tasks (Synth), and code tasks (Code). Rel. denotes the average score relative to the full KV cache.
Method	S-QA	M-QA	Summ	Fewshot	Synth	Code	Avg	Rel.
Full KV cache
Llama-3.1-8B-Instruct	44.6	47.6	29.2	54.0	55.1	48.1	46.4	100.0%
KV budget ratio = 0.125
KVzip	42.7	43.9	27.7	61.9	44.0	47.7	44.7	96.3%
KVzip + Ours	43.6	43.0	28.5	63.9	45.9	49.3	45.7	98.4%
KV budget ratio = 0.0625
KVzip	33.3	32.4	24.0	49.1	20.6	41.6	33.5	72.1%
KVzip + Ours	38.2	35.4	27.0	59.3	22.8	43.2	37.7	81.2%

Table E: RestoreKV training data statistics (context length in tokens).
Source	Samples	Ctx (mean)	Ctx range
LongAlpaca	2,488	7,306	4,961–15,237
PG-19	2,260	3,071	3,071–3,072
Tulu-3 FLAN	1,500	898	512–9,283
Total	6,248	4,236	512–15,237

Table F: Experimental environment. Hardware and software configurations used for all experiments.
Hardware
GPU	NVIDIA RTX PRO 6000 Blackwell (96 GB)
CPU	Intel Xeon Gold 6530
Memory	503 GiB
Software
OS	Ubuntu 22.04.5 LTS
Framework	PyTorch 2.8.0 (CUDA 12.8)

Table G: Sensitivity to Training Seeds. RULER-4K accuracy on Qwen3-4B, reported as mean±std over three training seeds.
	KV budget ratio
	0.40	0.20	0.10	0.05
KVzip	93.46	91.38	80.08	38.23
RestoreKV	94.25±0.04	93.37±0.11	88.21±0.51	72.81±0.57

Table H: Evaluation metrics used for each benchmark.
Benchmark	Evaluation metric
RULER-4K	Official string-matching score
QASPER	Maximum token-F1 over reference annotations
QuALITY	Answer-letter accuracy
LongHealth	Answer-letter accuracy
LongBench	Official per-task metric (F1, ROUGE-L, or accuracy)
SCBench	Official task-specific evaluator

Findings

On Qwen3-4B at a 5% KV budget, RestoreKV raised KVzip's RULER-4K accuracy from 38.2 to 73.2, and RestoreKV+ raised KVzip+'s from 51.6 to 70.7 under the same total budget.
Applied to KVzip+, RestoreKV reached 86.4 RULER accuracy at 16x compression on the KVPress Benchmark, with under 0.5% extra construction time and 84MB (0.4%) extra memory at a 32K-token context.
Applied to the learned scorer Fast KVzip, RestoreKV raised RULER-4K accuracy from 46.7 to 79.3 at 16x compression, exceeding plain KVzip's 59.0.
Against the synthesized-cache baseline AM-fast at r=0.05, RestoreKV+ scored 70.7 vs. 52.8 on RULER-4K while building a 4K-context cache in 0.74s versus 9.68s, about 13x faster.
Layer-wise analysis showed average context attention mass dropped to 0.56% under KVzip eviction, which RestoreKV recovered to 1.30%, and reduced KL divergence from the full-cache model from 0.198 to 0.167.

Where it can be used

Document QA or summarization services that compress a long context once and answer many subsequent questions from that cached representation, aiming to reduce quality loss from tight memory budgets.
Adding on top of existing KV eviction pipelines (KVzip, KVzip+, SnapKV, H2O) without retraining the underlying selection method itself.
Combining with lightweight learned scorers such as Fast KVzip to retain their speed advantage while reducing accuracy loss from aggressive compression.

Limits and open work

Performance recovery drops substantially when the eviction method used at inference differs from the one used during training, suggesting retraining may be needed if the base compression method changes.
Experiments cover Qwen3 models (0.6B-8B) and Llama-3.1-8B-Instruct with context lengths mostly up to about 15K-32K tokens plus one ~104K-token SCBench evaluation; larger models or other architectures were not tested.
Training data is limited to about 6.2k context-query pairs drawn from LongAlpaca, PG-19, and Tulu-3 Flan, so generalization to other document domains is not separately verified.
The default configuration (8 restore tokens, specific LoRA rank) was fixed for most experiments, so optimality of these settings in other contexts is not fully explored.

Why it matters

Services that cache long documents once and reuse them for many future questions face a direct tradeoff between memory savings and answer quality under aggressive compression, and this method reduces that loss without redesigning the compression pipeline itself. Because it plugs into existing eviction methods with a small trained add-on, it lowers the practical barrier to adopting tighter KV budgets in production systems.

Terms in this paper

KV cache · The stored key-value computations from previous tokens that let a transformer avoid recomputing them for later tokens
query-agnostic eviction · Compressing a context once before knowing future questions, then reusing that compressed cache for any of them
LoRA · A technique that adapts a frozen model by training only small low-rank matrices added to it, keeping cost low
self-distillation · Training a compressed-cache model (student) to match the outputs of the same model using the full, uncompressed cache (teacher)
RULER-4K · A synthetic benchmark of 13 tasks testing retrieval, tracing, and aggregation within a 4K-token context

Original abstract (English)

Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complements this selection-based formulation with learned restoration under the same total KV budget. Our key insight is that, although the information lost through eviction is context-specific, the mechanism for generating its

Authors · Changwoo Baek

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Changwoo Baek et al., arXiv:2608.01247, arxiv-nonexclusive