工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV

arXiv:2607.236932026-07-25

AI即便删掉了原始信息,依然能答对——因为那个值早已悄悄渗入了留下的另一句缓存里

长对话AI代理为节省内存,会丢弃部分缓存(KV缓存)只保留一部分。这项研究发现,即便彻底删除了写下某个事实的原始句子,后面一句提到它却没说出具体值的句子,其缓存中已经吸收了那个值,模型依然能答对。故意设计的、不直接说出答案的引导句可以把这种隐藏恢复率从6%提升到51%,但真实对话中自然出现的提及则没有可靠效果。

METAL LAB 解读图

被删除的事实如何渗入后续句子的缓存

证据状态已报告实测结果

  1. 第一步:来源事件早期句子如'寄存器S在线'设定了一个值,模型处理对话时将其编码进缓存
  2. 第二步:根节点事件后续句子'M对应S'引用了来源却没说出具体值,其缓存在处理过程中吸收了S的状态
  3. 第三步:服务时删除为节省内存,来源句子被丢弃,只保留根节点句子;通过供体对比较仅被删除值不同的两种场景
  4. 第四步:提问与回答询问M的状态时,尽管呈现的文本中没有任何地方说明该值,模型仍以99:0的比例答出了被删除的值
  5. 第五步:主动利用该机制刻意设计的引导句能把这种隐藏恢复率从6%提升到51%,而自然出现的提及效果则十分有限
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 长对话AI代理为节省内存会丢弃大部分缓存(KV缓存),只保留一小部分,但这一做法的前提——被保留的条目在其来源被删除后依然有用——此前几乎未被直接检验,本研究对此进行了检验。
  2. 研究者先写下'寄存器S在线',随后写一句从不说明具体值的'M对应S',删除第一句、只保留第二句后询问M的状态,Qwen3-8B在99个相关案例中全部(99:0,p=3.2×10⁻³⁰)答出了被删除的值,尽管没有任何呈现的文本说明该值。
  3. 刻意设计的、不说出答案的指令句(例如'现在判定并记录下来')把这种隐藏值恢复率在Qwen3-8B上从6%提升到51%,而从真实长期对话(REALTALK、LoCoMo)中自然采集的提及则没有检测到明显效果。
  4. 这种机制能携带的信息很有限:二元状态(是/否)恢复率远高于随机水平(93.4% 对 50%),但四选一、八选一的恢复率却接近随机水平,三位数字则从未被精确恢复。
  5. 在三个模型(Qwen3、Gemma-4、Ministral-3)上测试了十六种不同句式,发现没有一种句式在所有模型上都可靠有效——哪种表达能'写入'隐藏状态高度依赖具体模型。
Figure 1: Semantic materialization, and the comparison that isolates it. A: during one global prefill, a downstream root event (“M mirrors S”, which names no value) attends to the source that set S, so its cache rows come to encode S’s state — the root becomes a donor-dependent representation of S. B: at serve time we omit the source and serve the root — alongside a fixed donor-invariant decoy — and the query. A donor pair holds every served token and position fixed and flips only the omitted source; the answer follows that omitted value, so the served rows carry more than their visible text. C: the reference chain — the root mirrors the source and carries its value, while a reference edge mirrors the root, one hop further from the source, and mostly routes a query back to it.
Figure 1: Semantic materialization, and the comparison that isolates it. A: during one global prefill, a downstream root event (“M mirrors S”, which names no value) attends to the source that set S, so its cache rows come to encode S’s state — the root becomes a donor-dependent representation of S. B: at serve time we omit the source and serve the root — alongside a fixed donor-invariant decoy — and the query. A donor pair holds every served token and position fixed and flips only the omitted source; the answer follows that omitted value, so the served rows carry more than their visible text. C: the reference chain — the root mirrors the source and carries its value, while a reference edge mirrors the root, one hop further from the source, and mostly routes a query back to it.
Table 1: Donor-sensitive answers overwhelmingly follow the hidden donor. Follow:anti counts among scorable donor-dependent pairs (differing binary outputs), source omitted, across the serving cells and models defined in the text. Exact two-sided sign tests; Gemma-4 read out under the menu protocol.
CheckpointServing cellfollow : anti
Qwen3-8B (2025)root only99:0
Qwen3-8Bmention absent52:1
Ministral-3-8B (2025)mention absent90:1
Gemma-4-12B (2026)mention co-served80:0
Figure 2: Roots carry the dominant donor-aligned signal; served reference edges mostly route it (sparse serving, source omitted). A: the root-donor follow/anti rates (Wilson 95% CIs) in all four cells — mention-absent and mention-co-served, per checkpoint. Bars are rates over the stated denominators, not counts: mention-absent cells contribute one root contrast per base item (n=128/187), co-served factorial cells two (n=252/364), so heights are comparable only as rates. The mention’s own donor is inert on Qwen3 but has a substantial donor effect on Gemma-4. B: with a true referencing edge (“T mirrors M”) served, answers about T follow the root donor 41:0 on Gemma-4-12B; the edge alone carries far less (Gemma-4 2:0; weaker edge-local carryover on Ministral-3, 33:16). Panel A’s four cells still differ in serve set and appended query position, so cross-cell comparisons remain descriptive.
Figure 2: Roots carry the dominant donor-aligned signal; served reference edges mostly route it (sparse serving, source omitted). A: the root-donor follow/anti rates (Wilson 95% CIs) in all four cells — mention-absent and mention-co-served, per checkpoint. Bars are rates over the stated denominators, not counts: mention-absent cells contribute one root contrast per base item (n=128/187), co-served factorial cells two (n=252/364), so heights are comparable only as rates. The mention’s own donor is inert on Qwen3 but has a substantial donor effect on Gemma-4. B: with a true referencing edge (“T mirrors M”) served, answers about T follow the root donor 41:0 on Gemma-4-12B; the edge alone carries far less (Gemma-4 2:0; weaker edge-local carryover on Ministral-3, 33:16). Panel A’s four cells still differ in serve set and appended query position, so cross-cell comparisons remain descriptive.
Table 2: The sixteen-construction write bank, in two families. Mirror constructions read “register M ⟨relation⟩ register S”; flag constructions read “the alert flag for sensor r ⟨relation⟩ the check result” — neither naming a value. Write-through rate and class per model (serve carrier rows only, source omitted; 64 donor items each; comprehension mean .98, minimum .77). No construction is class W on all three models, and no Gemma-4 construction reaches W under this trace-generation readout; totals (W/P/U) are 5/4/7 Qwen3, 7/2/7 Ministral-3, 0/2/14 Gemma-4.
constr.relationQwen3Ministral-3Gemma-4
Mirror family — “register M ⟨relation⟩ register S”
followsfollows.83 W.81 W.55 U
mirrorsmirrors.80 W.89 W.52 U
copiescopies.78 W.77 W.55 U
shadowsshadows.78 W.95 W.50 U
kept_eqis kept equal to.72 P.84 W.58 U
matchesmatches.64 P.94 W.67 P
trackstracks.55 U.45 U.66 P
syncedis synchronized with.45 U.48 U.56 U
Flag family — “the alert flag for sensor r ⟨relation⟩ the check result”
consistwas made consistent with.94 W.95 W.47 U
inlinewas updated in line with.69 P.56 U.42 U
accordwas set according to.64 P.62 P.45 U
recordnow records.58 U.53 U.45 U
reflectwas set to reflect.52 U.64 P.53 U
updatewas updated accordingly.52 U.52 U.44 U
writewas written from.52 U.56 U.41 U
assignwas assigned by threshold.48 U.53 U.47 U
Figure 3: Native recovery is strong for binary state and falls toward chance as payload cardinality grows; exact numeric recovery fails. A–B: Qwen3-8B under one menu-form readout with independent decoys. A: harvest-after-drop accuracy by payload type — binary well above chance, four- and eight-way toward it, three-digit at zero. B: a write-time compute note shifts the copied→derived frontier but not numeric payloads. C: the generation–logit gap on selected legacy model–construction pairs (Llama-3.1, Gemma-2), where free generation under-reports the candidate-logit readout.
Figure 3: Native recovery is strong for binary state and falls toward chance as payload cardinality grows; exact numeric recovery fails. A–B: Qwen3-8B under one menu-form readout with independent decoys. A: harvest-after-drop accuracy by payload type — binary well above chance, four- and eight-way toward it, three-digit at zero. B: a write-time compute note shifts the copied→derived frontier but not numeric payloads. C: the generation–logit gap on selected legacy model–construction pairs (Llama-3.1, Gemma-2), where free generation under-reports the candidate-logit readout.
Table 3: The answer follows the root donor on both checkpoints. Source-omitted 2×2 donor factorial (root donor × downstream-mention donor), answer about the mirror register; per-item paired effects on P⁡(answer=ONLINE), complete-case (n=126 Qwen3, free-generation readout; n=182 Gemma-4, menu). Interaction = mention effect at root=ON minus at root=OFF. †90% TOST interval; others 95%. The two root columns are two different contrasts, not one measurement repeated. They are the mention-absent root-only contrast (n=128/187) and the root main effect with the mention co-served, measured under serving protocols that also differ in the appended query’s position. Estimator and ITT sensitivity: App. B; full four-cell counts: Table 4.
Root effect
Checkpointmention-absentco-servedMention effectInteraction
Qwen3-8B+.398 [.311,.486]+.083 [.014,.153]−.028 [−.073,+.017]†+.008 [−.079,+.095]
Gemma-4-12B+.091 [.050,.132]+.220 [.175,.265]+.159 [.121,.197]+.011 [−.070,+.092]
Figure 4: Passive natural mentions are not a dependable write interface; deliberate carriers establish the primitive in controlled trajectories. A: on real long-term dialogs (REALTALK, LoCoMo), harvesting natural mentions yields no end-to-end benefit over isolated encoding; Gemma-4 is served natively (original positions, key-masked — the geometry an eviction-style system presents, App. D), where its LoCoMo cell is equivalent to isolated encoding under the ±.05 band. B: Qwen3-8B donor-follow rate across the five carrier arms (passive mention to explicit-text upper bound), 95% CIs; the three-recent-model profiles — construction-sensitive on Qwen3, saturating on Gemma-4, abstention-dominated on Ministral-3 under free generation, with explicit text recovering everywhere — are in Table 7.
Figure 4: Passive natural mentions are not a dependable write interface; deliberate carriers establish the primitive in controlled trajectories. A: on real long-term dialogs (REALTALK, LoCoMo), harvesting natural mentions yields no end-to-end benefit over isolated encoding; Gemma-4 is served natively (original positions, key-masked — the geometry an eviction-style system presents, App. D), where its LoCoMo cell is equivalent to isolated encoding under the ±.05 band. B: Qwen3-8B donor-follow rate across the five carrier arms (passive mention to explicit-text upper bound), 95% CIs; the three-recent-model profiles — construction-sensitive on Qwen3, saturating on Gemma-4, abstention-dominated on Ministral-3 under free generation, with explicit text recovering everywhere — are in Table 7.
Table 4: Four-cell counts for the source-omitted 2×2 donor factorial (Table 3): full per-cell ONLINE/OFFLINE/other tallies over all items (n=128 Qwen3, 192 Gemma-4), so the other (non-binary) readouts are visible rather than conditioned away; the effects in Table 3 use the complete-case subsets (n=126/182), with an ITT re-analysis reported there. The last two columns are the mention-absent cells; note their root contrast is measured with the query closer to the root (App. G). Root donor sets the queried register’s state; the mention donor is the value-free downstream reference. Qwen3-8B free-generation readout; Gemma-4-12B menu readout.
root=ONroot=OFFmention absent
Checkpointmen=ONmen=OFFmen=ONmen=OFFroot=ONroot=OFF
Qwen3-8B64/63/167/60/153/74/157/71/082/46/031/97/0
Gemma-4-12B121/63/889/98/579/109/450/139/3103/88/182/106/4
Figure 5: Cross-generation construction bank (exploratory): Qwen3 as a recent anchor and four 2024 legacy models. Sixteen constructions × five models (Qwen3/Gemma-2/Qwen2.5/Llama/Mistral). A: KV write-through (serve carrier rows only, source omitted), class W/P/U/X. B: full text visible — near-uniformly solved except Mistral. n per cell: 64/48/64/48/32.
Figure 5: Cross-generation construction bank (exploratory): Qwen3 as a recent anchor and four 2024 legacy models. Sixteen constructions × five models (Qwen3/Gemma-2/Qwen2.5/Llama/Mistral). A: KV write-through (serve carrier rows only, source omitted), class W/P/U/X. B: full text visible — near-uniformly solved except Mistral. n per cell: 64/48/64/48/32.
Table 5: X8 same-path injected-carrier positive control (§7): gold recovery with an injected donor-paired carrier vs. isolated encoding of the same carrier text, routed through the identical harvest/splice/recognition path (n=62 QA each, REALTALK+LoCoMo). Both rows use the identical 62-item set over 17 conversations. Both checkpoints are served here through the compact path, so for Gemma-4 this control speaks to the splice/recognition path rather than to the native geometry of Table 6. Paired difference with McNemar exact p and discordants (injection-helps:injection-hurts), plus a conversation-clustered bootstrap 95% CI (seed-pinned B=4000, matching Table 6); the McNemar p is item-level, and clustering changes neither conclusion. Qwen3’s forced choice is option-prior-saturated (a counterfactual carrier still selects gold ∼80%), so its control cannot fire. Gemma-4 does separate (+.194, clustered CI excludes zero): a detected end-to-end contextual benefit, whose +12 gold gain is exactly matched by twelve fewer abstentions (None 42→30; wrong unchanged at 3). The donor-flipped counterfactual arm was collected but fires on too few items to adjudicate — Gemma-4 3:0 follow:anti with 33/62 counterfactual readouts unscorable, Qwen3 1:0 — so this does not separate state-specific transfer from generic readout activation, and we report it as a partial positive.
Checkpointinj. goldiso. goldΔMcNemar p (disc.)clustered 95% CI
Qwen3-8B50/6251/62−.0161.00 (3:4)[−.102,+.067]
Gemma-4-12B29/6217/62+.194.043 (21:9)[+.030,+.357]
Table 6: Real-dialog passive harvest (harvested − isolated recognition accuracy; per-question paired, conversation-clustered bootstrap, seed-pinned B=4000; n = qualifying QA, clu = conversation clusters). Designated equivalence to 0 uses a ±.05 TOST on the 90% interval. Both checkpoints are evaluated on the identical qualifying sets (497 REALTALK / 564 LoCoMo QA over 10 conversations each). Qwen3 shows no benefit on REALTALK and a marginal negative on LoCoMo (90% excludes 0, 95% does not), and reaches ±.05 equivalence on neither — so we log “no advantage detected,” not equivalence. The Gemma-4 rows are served natively — retained rows at their original positions, omitted rows key-masked, query at the original end position — which is the geometry an eviction-style serving system presents (App. D); they are read by candidate logit, as elsewhere in this paper. Under that geometry Gemma-4 harvesting on LoCoMo is equivalent to isolated encoding within the ±.05 band — the only cell here attaining designated equivalence rather than an undetermined null. †REALTALK’s ten dyads form three participant components, so its intervals are descriptive and support no cluster-based inference; formal verdicts are LoCoMo-only. A legacy Gemma-2 replication (negative) is in Appendix C.
ModelDatasetn/cluharv−iso90% CI95% CIverdict
Qwen3-8BREALTALK†497/10−.040[−.094,+.020][−.104,+.032]descr.
Qwen3-8BLoCoMo564/10−.044[−.081,−.005][−.088,+.005]inconcl.
Gemma-4-12BREALTALK†497/10+.012[−.002,+.025][−.006,+.027]descr.
Gemma-4-12BLoCoMo564/10−.011[−.024,+.002][−.026,+.004]equiv.
Table 7: An answer-free compute directive lifts donor-aligned recovery from 11/192 under a passive mention to 97/192 on Qwen3-8B. X9 donor-transition classes on the subject-addressed readout under harvested serving (follow/anti/const/other; 192 donor pairs per arm per model). exp carries the answer in text (upper bound). Three recent-model profiles: Qwen3 is construction-sensitive with a non-saturating latent positive (dir−pas +.172 CI [.109,.234], p=2.5×10−7; nld−pas +.448, p=5.8×10−25); Gemma-4 materializes every construction to ceiling; Ministral-3 is abstention-dominated in every latent arm (no reliable donor-aligned advantage), recovering only explicit text. Readouts. They follow each model’s free-generation reliability: Qwen3 and Ministral-3 via free-generation trace, Gemma-4 via the menu readout it requires (§4). Legacy 2024 columns (Gemma-2, Qwen2.5) are in Appendix C.
ArmQwen3-8BGemma-4-12BMinistral-3-8B
follow/anti/const/other of 192
pas passive mention11/0/181/0189/0/3/00/0/4/188
bind grounded binding17/0/175/0191/0/1/00/0/0/192
dir structured directive44/0/148/0191/0/1/06/3/64/119
nld NL compute directive97/0/95/0188/0/4/04/4/71/113
exp explicit text192/0/0/0192/0/0/0182/0/0/10
Table 8: X10 serve-set ablation on the three recent models (Qwen3, Gemma-4, Ministral-3 — three families; donor-follow rate serving the carrier row alone, the shared downstream-review row alone, or both — c/r/b; 192 donor pairs per serve-set cell per model; same manifest/seeds/menu readout; retained decoy and unrelated-register rows precede the source and are donor-invariant; full follow/anti/const/other counts in Table 9). Retained rows keep their original absolute positions; the query is appended after the last served row, so — as the review is always the trajectory’s final row — r and b share the query position while c places it earlier. On these checkpoints the carrier carries the dominant donor-aligned signal: under the query-position-controlled contrast r→b, adding the carrier drives Qwen3 follows 0→.42 (nld; review alone shows no detected effect), and Gemma-4 saturates the carrier on every arm. The c-vs-b contrast also moves the query, so the co-served-review change (nld .19→.42) is an interface-level serve-set effect, not a span interaction. Gemma-4 keeps a weak donor-aligned review-local signal (review-only nld/exp 18:0/15:0 follow:anti); Ministral-3 is abstention-dominated on latent arms, its explicit-text review row is donor-sensitive but anti-aligned (0:18; Table 9) — not inert. Ministral-3’s passive-carrier drop under co-serving (pas .09→.00) is not identified: the query-controlled r→b contrast shows no detected change (1→0, p=1.0), so the cause (query offset vs. review membership) is unresolved. A review-dominant shift, the review leading, appears only in the exploratory Gemma-2 run (App. C) and is not reproduced on any recent model.
Qwen3-8BGemma-4-12BMinistral-3-8B
Armcrbcrbcrb
pas.00.00.03.98.01.99.09.01.00
bind.06.00.09.99.04.99.00.02.00
dir.21.00.251.00.071.00.00.04.03
nld.19.00.42.99.09.98.00.02.02
exp.97.001.001.00.081.00.98.00.98
Table 9: X10 full transition counts (follow/anti/const/other; 192 donor pairs per cell) for the three recent models, serving carrier-alone / review-alone / both. const = donor-blind committed answer; other = at least one unscorable donor output — a donor-blind identical abstention, or a discordant pair where exactly one side abstains (donor-dependent but undirected). The interpretable, query-position-controlled contrast is review→both (query fixed at the trajectory’s final review row): adding the carrier drives Qwen3 nld follow 0→81 (McNemar 81:0, exact p=8×10−25; dir 0→48), replicated 0→98 under a disjoint seed (n=192; maximum per-cell follow-rate deviation .09/.03/.05 across the three models). Gemma-4’s review-only keeps a weak donor-aligned signal (nld 18:0) and Ministral-3’s explicit-text review is anti-aligned (0:18, replicated 2:21).
ModelArmcarrierreviewboth
Qwen3-8Bpas0/0/192/00/0/192/05/0/187/0
bind12/0/180/00/0/192/017/0/175/0
dir41/0/151/00/0/192/048/0/144/0
nld36/0/156/00/0/192/081/0/111/0
exp187/0/5/00/0/192/0192/0/0/0
Gemma-4-12Bpas188/0/4/02/0/166/24191/0/1/0
bind190/0/2/07/0/155/30191/0/1/0
dir192/0/0/013/0/169/10192/0/0/0
nld191/0/1/018/0/58/116189/0/3/0
exp192/0/0/015/0/154/23192/0/0/0
Ministral-3-8Bpas18/0/17/1571/2/46/1430/1/6/185
bind0/0/20/1724/4/46/1380/0/0/192
dir0/0/22/1707/8/115/625/8/67/112
nld0/1/2/1893/4/83/1023/4/80/105
exp189/0/0/30/18/36/138188/0/0/4
Table 10: Selected cross-model panel (recent + legacy; Qwen2.5 omitted for space — it ran the construction bank and X9). Bank W = W-class constructions under the trace protocol (of 16); probe = candidate-logit reachability; edge = follow:anti among donor-sensitive pairs, ask-through-edge, and the parenthetical is the same run’s root-only bridge cell (serve {decoy, root}, ask the mirror register) under the same readout († = menu readout, which Gemma-4’s free-generation interface requires; unmarked cells free-generation). The §3 headline 99:0 is a separate root-only run under free generation, not this run’s root cell. X9 latent = best latent-carrier donor-follow rate (abst. = abstention-dominated, no reliable latent positive); X9 expl. = explicit-text follow rate; X10 serve-set results are in Table 8. Legacy rows (2024) are exploratory.
ModelYearBank WProbeEdge (root)X9 latentX9 expl.
Mistral-7B2024excl. (13 X)————
Gemma-2-9B20248/168/16 ≥.90174:0.531.00
Llama-3.1-8B20240/16split———
Qwen3-8B20255/16≈gen84:0† (root 89:0†).511.00
Ministral-3-8B20257/16—298:0abst..95
Gemma-4-12B20260/16≥.90 on 8/1641:0†≈1.01.00
Table 11: Legacy active-materialization counts (exploratory; supports no headline). Left: X9 donor-transition (follow/anti/const/other, 192 pairs/arm). Right: Gemma-2 serve-set ablation (carrier/review/both donor-follow rate).
X9 countsGemma-2 X10
ArmGemma-2Qwen2.5crb
pas85/0/107/00/0/61/131.32.24.55
bind14/0/178/00/1/21/170.03.46.09
dir0/0/192/00/0/46/146.01.42.01
nld101/0/88/30/0/74/118.44.09.31
exp192/0/0/0190/0/0/21.001.001.00
Table 12: Compact vs. native serving on Gemma-4 as the far row’s original distance from the query crosses the 1024 window. gap = query-to-far-row distance in tokens. Within one window the two paths agree (aggregate follow counts identical; first-token argmax 29/32 on the ONLINE-donor arm); beyond it they diverge sharply. The compact path does not simply over-expose the far row — its position/slot discontinuity attenuates it — but the served numbers are not native.
gap (tok)windownative follow:constcompact follow:constargmax agree; max|Δ​logit|
∼200within22:1022:1029/32; 1.16
∼1590beyond14:186:264/32; 13.9
∼3430beyond6:25 (+1 anti)3:2910/32; 13.5

研究结果

  • 在Qwen3-8B上,初次实验中答案跟随被删除来源的值达99:0,256对复现实验中为130:0,没有出现相反方向的情况
  • 在三个模型上测试的16种句式构造中,没有任何一种句式在所有模型上都可靠有效;各构造的写入成功率从随机水平到95%不等,因模型而异
  • 二元状态恢复率为93.4%,远高于50%的随机基线,而四选一(22.3%)和八选一(15.6%)的恢复率接近各自的随机水平,三位数字在192次测试中从未被精确恢复
  • 故意设计的指令句将Qwen3-8B的隐藏值恢复率从6%提升到51%,但从真实长对话中自然采集的提及未检测到明显收益
  • 在Gemma-4-12B的滑动窗口缓存中,按原始位置保留条目与将条目重新打包相邻放置相比,在34%的测试项目上给出了不同的判断结果,说明缓存的排布方式本身也会影响结果

可应用场景

  • 评估长对话AI代理的缓存淘汰策略时,不仅要看下游准确率,还要检查被删除的信息是否已经渗透进保留的缓存条目中
  • 在对话中刻意插入不直接说出答案、但引导模型进行判断并记录的句子,以便日后即便原始证据被删除也能恢复重要状态
  • 对已记忆状态进行更正时,采用追加新事件(打补丁)的方式而非直接编辑缓存
  • 在看到'删除来源后准确率没有下降'的结果时保持谨慎解读,因为该来源的信息可能早已被吸收到其他地方

局限与待验证事项

  • 较强的恢复效果主要在受控的合成对话场景中得到验证;在真实长期对话(REALTALK、LoCoMo)中自然出现的提及并未显示出可靠的收益
  • 这种方式能携带的信息种类有限:多选项和精确数字信息大多无法恢复,不同于简单的二元状态
  • 哪种句式能触发该效应高度依赖具体模型,没有一种构造在三个测试模型上都通用,需要针对每个模型单独校准
  • 在使用滑动窗口的模型(Gemma-4)上,保留条目按原始位置存放还是重新打包排布会导致不同结果,因此结果会受具体服务系统缓存排布方式的影响
  • 研究覆盖了三个较新模型(Qwen3、Gemma-4、Ministral-3)及部分旧模型,且均使用未经微调、无额外训练读出器的冻结模型,尚未验证在其他架构或训练方式下的普适性

为什么重要

这意味着删除一个来源事件后若没观察到准确率下降,并不能证明该来源是多余的——它的信息可能早已渗入了被保留的条目中。设计长期记忆系统或缓存淘汰策略的人,需要把这种隐藏信息转移现象纳入考虑,而不能只依赖准确率评估。

本文术语

  • KV缓存 · 模型处理对话时保存的中间计算结果,复用它可以避免重新处理全部历史内容
  • 语义物化(semantic materialization) · 已删除事实的值最终渗透进后续一句看似无关的缓存内容中的现象
  • 供体对(donor pair) · 两个除被删除来源的值不同、其余完全相同的场景,用来检验答案是否跟随那个被删除的值
  • 驱逐(eviction) · 为节省内存而删除较旧或看起来不重要的缓存条目的做法
  • 根节点/引用边 · 根节点是直接引用来源并吸收其值的句子,引用边则是指向根节点、大多只是把问题'转发'回根节点的更远一层句子

论文原文摘要（英文）

Long-horizon agents increasingly reuse their KV cache as memory: a serving system keeps a subset of cached entries and drops the rest. Eviction and episodic-memory schemes therefore rest on a premise rarely tested directly, that a retained event is still informative once the observations that produced it are gone. We test it by omitting one earlier observation from what is served, across otherwise identical agent histories. Among items sensitive to that observation, the answer overwhelmingly fol

作者 · Zefeng Cai

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Zefeng Cai et al., arXiv:2607.23693, CC BY 4.0