ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
arXiv:2608.196622026-08-21
Making AI agents stop re-reading the same tool manuals every time, for a 3x-plus speed-up
AI agents that call external tools have to re-encode tool and skill descriptions ('resources') from scratch whenever they show up in a new combination or order, which breaks standard KV-cache reuse. ReCache fixes this by letting each resource be cached independently and by trimming down what gets stored to only the parts that actually matter for making the right call. The result is up to 3.655x faster first-token response and 92.43% less cache memory, with accuracy nearly unchanged from the uncompressed baseline.
What they did
Problem: tool/skill descriptions ('resources') recur across requests in different combinations and orders, so standard prefix caching, which only reuses identical starting text, can't reuse their computed key-value (KV) states
Fix 1 - Resource-wise attention: removes cross-talk between different resources and resets each resource's internal position numbering from zero, producing KV blocks that stay valid regardless of order or combination
Fix 2 - Structural pruning: measures which transformer layers and KV head groups actually contribute to correct tool invocation, then only lets those selected 'routes' access resource information, blocking the rest
Fix 3 - Semantic pruning: keeps only the invocation-critical text, resource name, argument names, argument descriptions, and a final summary token, discarding the rest
Results on a benchmark built from seven public tool/skill-use datasets: resource-wise attention alone nearly matches full dense-attention accuracy (82.3% vs 82.4% Inv-F1) while giving a 3.655x speed-up in time-to-first-token; the full ReCache framework cuts allocated KV-cache memory by 92.43% and speeds up attention computation by 1.423x
Figure 1: Cross-resource attention on Qwen3-4B.
Table 1: Selected structural budgets. CC and ρ are percentages; ΔInv-F1 is measured relative to Ωfull after full-data training.
Model
KL/L
KG/G
CCL
CCG
ρ
ΔInv-F1 (%)
Ql
20/36
3/8
97.7
47.3
79.2
−0.2
Qs
20/28
7/8
99.8
99.3
37.5
−0.7
(b) Attention from subsequent conversation tokens onto resources, normalized and aligned by relative position.
Table 2: Resource-wise attention ablation.
Method
Inv-F1
ID-F1
Halluc.
TTFT (ms)
Dense
82.4
96.0
0.0
26.319
Block
82.2
95.8
0.1
–
Ωfull
82.3
96.0
0.2
7.200
Figure 2: ReCache constructs reusable KV blocks for each Ri and progressively reduces their subsequent cache retention and access. (a) Resource-wise attention removes inter-resource attention and resets positions from zero within each Ri. (b) Structural pruning exposes resource KV states (using Mresource) only through selected layer (Li)–KV-head-group (Gi) routes Ω⋆ (highlighted in blue). The remaining routes use Mcontext. (c) Semantic pruning retains resource names, argument (arg) names and descriptions (desc.), and the final suffix token. The final mask combines these decisions, ensuring that Y attends exclusively to retained semantic fields within assigned routes during decoding.
Table 3: Results on 𝒯IND. Efficiency is reported relative to Dense. SMP denotes Semantic Pruning.
Method
Attn. ×
Mem. ↓
Inv-F1
ID-F1
Halluc.
Dense
–
–
82.4
96.0
0.0
Ωfull
1.001×
0.47%
82.3
96.0
0.2
Structural pruning
Ω20,G
1.016×
44.71%
82.5
95.7
0.3
Ωfull + SPEED
1.013×
44.71%
79.3
92.9
2.8
Ω20,3
1.314×
79.27%
82.1
95.6
0.2
Ωfull + SA20,3
1.302×
79.27%
79.1
93.7
1.0
Semantic pruning
Ωfull + Gist
1.029×
99.22%
39.2
46.9
51.4
Ωfull + Beacon
1.020×
75.42%
78.6
92.5
1.4
Ωfull + SMP
1.021×
63.57%
81.6
95.2
0.2
ReCache
1.423×
92.43%
80.3
94.9
0.2
Figure 3: Structural contribution coverage in Ql and Qs. Cumulative leave-one-in contributions of layers (top) and KV head groups (bottom); vertical lines mark the selected budgets.
Figure 4: Efficiency on varying resource lengths Dℛ. Requests are grouped by average resource length into Small (0≤Dℛ<1K), Medium (1K≤Dℛ<5K), Large (5K≤Dℛ<10K), and XL (Dℛ≥10K). SMP denotes semantic pruning.
Table 5: Dataset composition and split statistics. Source rows report eligible examples before final selection; the Toucan aggregate is not double-counted in the summary row. Distinct names counts source-level unique declared tools or skills, Subset dup. counts records beyond the first occurrence of an identical candidate configuration, and Recurring names counts resources appearing in multiple records. The summary row corresponds to 18.8% duplicated candidate records and 77.3% recurring resource names. IND cand. and OOD cand. denote eligible in-distribution and out-of-distribution candidates, and dashes mark unavailable statistics.
Group
Source
Train pool
Distinct names
Subset dup.
Recurring names
IND cand.
OOD cand.
Tool
ToolACE
8,400
14,566
178
7,144
200
195
APIGEN
10,000
2,806
734
2,805
500
146
Toucan (all generators)
13,071
4,355
6,901
4,355
600
493
- Kimi-K2
3,333
1,103
1,772
1,103
200
145
- GPT-OSS-120B
4,184
1,412
2,162
1,412
200
130
- Qwen3-32B
5,554
1,840
2,967
1,840
200
218
ToolRet
1,150
1,593
235
431
100
150
ToolMind
20,000
14,504
1,853
14,497
500
190
WildToolBench
256
–
–
–
–
–
Skill
SkillRouter
87
–
–
–
–
–
Sources with frequency statistics
52,621
37,824
9,901
29,232
1,900
1,174
Candidate pool
52,964
–
–
–
1,900
1,174
Final dataset
49,424
–
–
–
1,000
1,000
(b) TPOT
Table 6: Structural budget ablation for Qs across training data regimes with comparable budgets.
Config
Inv-F1
Resource ID
Halluc.
ID-P
ID-R
ID-F1
Ωfull
72.1
92.0
90.8
90.9
0.4
ΩL,5
49.6
74.5
73.5
73.4
4.8
ΩL,6
53.4
79.6
77.9
78.3
3.2
ΩL,7
57.3
83.4
81.8
82.1
2.9
Ω15,G
69.3
89.2
88.3
88.3
1.1
Ω20,G
70.5
90.6
89.5
89.6
1.1
(c) Attn.
Table 7: Structural budget ablation trained on 𝒟.
Config
Inv-F1
Resource ID
Halluc.
ID-P
ID-R
ID-F1
𝑸𝒔
Ωfull
80.7
94.9
94.3
94.4
0.1
Ω20,5
78.8
94.1
93.5
93.6
0.5
Ω20,7
80.0
94.7
94.0
94.1
0.3
𝑸𝒍
Ωfull
82.3
96.4
95.8
96.0
0.2
Ω20,2
81.9
95.3
94.7
94.7
0.3
Ω20,3
82.1
96.1
95.5
95.6
0.2
(d) Mem.
Table 8: Semantic pruning ablation trained on 𝒟subset. Arg stands for argument, Desc. stands for description. Ω20,3 denotes the routing configuration with KL=20,KG=3.
Config
Inv-F1
Resource ID
Halluc.
ID-P
ID-R
ID-F1
Ω20,3
76.1
93.1
92.8
92.5
0.9
Schema representation ablation
Suffix-only Schema
14.8
22.8
23.0
22.9
76.0
+ Resource Name
45.2
89.9
88.8
88.9
1.8
+ Arg Name
67.7
92.1
91.1
91.2
1.3
+ Resource Desc.
67.9
92.2
91.4
91.4
1.1
+ Arg Desc.
72.8
92.8
91.9
92.0
1.0
+ Both Desc.
73.4
93.0
92.2
92.2
1.2
Table 9: Individual contribution weights and cumulative contribution coverage of structural units. Within each panel, units are ranked independently by decreasing wi, and CC at rank k is the cumulative positive contribution weight through rank k. Bold entries fall within the empirical budgets KL=20,KG=3 for Ql and KL=20,KG=7 for Qs.
Layers
𝑸𝒍(Qwen3-4B)
𝑸𝒔(Qwen3-1.7B)
rank
unit
𝒘𝒊 (%)
CC (%)
rank
unit
𝒘𝒊 (%)
CC (%)
1
L34
15.56
15.6
1
L22
20.09
20.1
2
L35
10.53
26.1
2
L25
12.90
33.0
3
L23
9.37
35.5
3
L20
12.14
45.1
4
L25
8.31
43.8
4
L26
10.90
56.0
5
L32
8.01
51.8
5
L27
10.39
66.4
6
L30
6.95
58.7
6
L17
6.82
73.2
7
L33
6.89
65.6
7
L21
6.70
79.9
8
L27
5.17
70.8
8
L19
4.65
84.6
9
L21
4.87
75.7
9
L24
4.00
88.6
10
L28
4.64
80.3
10
L23
2.13
90.7
11
L24
4.02
84.3
11
L18
1.89
92.6
12
L22
3.74
88.1
12
L15
1.73
94.3
13
L17
2.30
90.4
13
L13
1.39
95.7
14
L31
1.96
92.3
14
L10
1.13
96.9
15
L18
1.29
93.6
15
L9
1.03
97.9
16
L14
1.26
94.9
16
L4
0.88
98.8
17
L20
0.78
95.7
17
L16
0.56
99.3
18
L36
0.69
96.3
18
L12
0.18
99.5
19
L19
0.68
97.0
19
L11
0.15
99.7
20
L16
0.68
97.7
20
L8
0.14
99.8
21
L10
0.60
98.3
21
L28
0.11
99.9
22
L5
0.33
98.6
22
L5
0.06
100.0
23
L11
0.29
98.9
23
L2
0.02
100.0
24
L9
0.27
99.2
24
L1
0.00
100.0
25
L12
0.26
99.5
25
L3
0.00
100.0
26
L15
0.26
99.7
26
L6
0.00
100.0
27
L13
0.13
99.8
27
L7
0.00
100.0
28
L8
0.07
99.9
28
L14
0.00
100.0
Why it matters
Tool-using AI agents currently re-process bulky tool descriptions on every request, which slows responses and inflates serving costs; ReCache offers a concrete way to cut that redundant computation while keeping performance intact. Its code is publicly released, so developers building agentic systems can adopt it directly.
Terms in this paper
KV cache · stored key-value computations from previously processed tokens that a language model reuses instead of recomputing
prefix caching · a caching method that only reuses prior computation when the new input starts with an identical sequence of text
TTFT (time-to-first-token) · how long it takes the model to produce its very first output token after receiving a request, a key latency metric
structural / semantic pruning · compression methods that remove less-important parts either at the model's layer/head level or at the text/token level
resource (tool/skill schema) · the structured text description of a callable tool or skill, including its name, arguments, and descriptions
Original abstract (English)
Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce \textbf{ReCache}, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3\% versus 82.4\% Inv-F1) while providing a 3.655$\times$ time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43\% and accelerates attention by 1.423$\times$. These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.
