One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Can Agent Memory Systems Track Evolving State?

arXiv:2608.196522026-08-21

AI agents keep answering with facts that were already updated, and this paper measures that failure directly

Memory systems for LLM-based agents have mostly been tested on whether they can find the right fact in a long conversation, not on whether they know that fact has since changed. This paper names that separate skill 'state tracking', builds a 234-scenario benchmark called StateMemBench to isolate it, and introduces StateMem, a method that explicitly tracks which facts have been superseded and which values depend on them. StateMem lifts accuracy by up to 1.8x over the strongest comparable baseline, and a lightweight wrapper version of the same idea boosts six existing memory systems as well.

What they did

  1. Existing benchmarks mainly check whether an agent can retrieve a relevant fact from a long conversation; this paper isolates a different failure, 'state drift', where the fact is right there in context but the agent still answers with a stale or already-overridden version of it
  2. They built StateMemBench (234 multi-session scenarios, 322 graded probes) with closed-pool grading that scores whether an answer reflects the current state, a superseded state, or is simply wrong
  3. They propose StateMem, which parses conversation turns into structured 'state units', marks values as superseded when they change, and automatically flags any dependent values for recheck through a dependency graph
  4. On DeepSeek-V4-Flash, StateMem improves current-state accuracy 1.8x over the strongest same-backbone baseline (0.205 to 0.363); on Qwen-3.5-9B it improves 1.6x over the strongest memory system (0.149 to 0.233), while staying competitive with feeding the whole conversation into context directly
  5. Applied as a lightweight single-call wrapper on top of six existing memory and retrieval systems, it raised current-state accuracy by 32 to 67 points, and a matched control experiment showed 15 to 32 of those points come from the state structure itself, not just from adding more text
Figure 1: StateMemBench targets state tracking: maintaining currently operative values of facts, rules, and derived quantities under cross-session revision, separable from recall.
Figure 1: StateMemBench targets state tracking: maintaining currently operative values of facts, rules, and derived quantities under cross-session revision, separable from recall.
Table 1: Cross-benchmark failure-mode distribution. Each cell is the share of that benchmark’s confirmed failures assigned to each bucket; the drift column is highlighted. †On LongMemEval oracle every gold fact is in the prompt by construction, so no failure there can be a retrieval failure. The small-n τ2-bench-Z result (n=16) is reported in §3.3. Labeling, adjudication, and per-row details: Appendix A.
Benchmarkndriftretr.comp.schemareasonFA
MemoryArena
Shop.20063.522.52.50.09.5
Search20038.525.511.017.54.5
Travel20019.011.046.521.00.0
LongMemEval oracle
LME†3644.40.00.025.02.827.8
Figure 2: StateMem has three stages: an ingestion stage in which a conversation turn is parsed into state units, an update stage that updates the value of existing units, and a test time stage where the agent draws upon the valid state to answer the question. Here, we show how StateMem would handle a finance-based scenario.
Figure 2: StateMem has three stages: an ingestion stage in which a conversation turn is parsed into state units, an update stage that updates the value of existing units, and a test time stage where the agent draws upon the valid state to answer the question. Here, we show how StateMem would handle a finance-based scenario.
Table 2: Comparison with agent-memory and long-horizon benchmarks with state-tracking evaluation. (✓) = partial satisfied. Scale is reported in each benchmark’s native unit. Column definitions and per-benchmark justifications are in Appendix C.
BenchmarkMulti-sess. dialogueScale (native unit)Updates centralSuper- sessionVerifiable goldDrift scoredAnti-update controlsPaired horizons
LoCoMo (20)∼300 turns
LongMemEval (30)∼40–80 sess. (S)(✓)(✓)
MemoryAgentBench (11)(✓)varies(✓)(✓)(✓)
MemoryArena (9)(✓)varies
STATE-Bench (21)450 tasks(✓)(✓)
StateMemBench (ours)∼600 turns
Table 3: State-agreement accuracy (gold rate) on StateMemBench: 190 in short scenarios (∼165 turns, one probe each) and 132 in 44 long fused scenarios (∼600 turns, three probes each), graded by a fixed deepseek-v4-pro judge. Memory systems (Mem0 (5), A-Mem (34), LightMem (7), MemoryOS (14), StateMem), retrieval baselines (BM25, text-embedding-3-small (22)), and graph-RAG systems (nano-graphrag, HippoRAG, LightRAG, GraphRAG) run on each substrate (Qwen-3.5-9B (25), thinking off; deepseek-v4-flash (33), thinking off); long-context baselines are raw models with full history in context. StateMem ablations (indented) excluded from best-marking. Per column within each panel, best bold, second best underlined. ***StateMem’s overall gold rate beats the strongest memory baseline on both backbones (paired McNemar test; both p < 0.001)
ConditionOverallShortLong
Long-contextQwen-3.5-9B0.1490.1370.167
Qwen-3.6-35B-A3B0.0970.0630.146
GPT-5.4-Nano0.2770.3110.227
DeepSeek-V4-Flash0.1490.1320.174
Qwen-3.5-9BNo memory0.0060.0000.015
BM250.1250.0950.167
Dense0.1300.1050.167
nano-graphrag0.1430.1260.167
HippoRAG0.1300.1050.167
LightRAG0.1150.0890.152
GraphRAG0.2240.2160.235
Mem00.1490.1160.197
A-Mem0.1270.1000.167
LightMem0.0190.0210.015
MemoryOS0.0250.0260.023
StateMem (ours)0.233***0.2370.227
– extraction-only0.1550.1260.197
– + supersession0.1710.1370.220
– w/o dep.-propagation0.2220.2020.250
– w/o recompute-guid.0.2080.1950.227
deepseek-v4-flashNo memory0.0030.0000.008
BM250.1430.1050.197
Dense0.2050.1680.258
nano-graphrag0.1830.1740.197
HippoRAG0.1840.1320.258
LightRAG0.0930.0580.144
GraphRAG0.1740.1370.227
Mem00.1770.1680.189
A-Mem0.1990.1580.258
LightMem0.0120.0210.000
Table 4: Accuracy on LongMemEval (full set, n=500, k=20) and LoCoMo (n=1,985), with each method run on both substrates (Qwen-3.5-9B; deepseek-v4-flash; thinking off) and all answers graded by a fixed deepseek-v4-pro judge. Per column, best in bold, second best underlined.
LongMemEvalLoCoMo
ConditionQwen-9BDS-V4Qwen-9BDS-V4
Long context0.5500.6660.6120.587
No memory0.0620.0560.0100.005
Mem00.5660.5940.4810.462
A-Mem0.5200.5100.2660.279
LightMem0.0640.0560.0270.020
MemoryOS0.0800.0720.0400.031
StateMem (ours)0.5800.6560.5660.592
Table 5: Closed-pool (n=322) drift-rate and in-pool engagement, methods that produce non-trivial in-pool counts on at least one backbone. In-pool (I-P) = method produced one of the closed-pool options. Drift-rate (D-R) = drift / n (errors are 0 for every method). StateMem’s I-P is bold.
Qwen-3.5-9BDeepSeek-V4-Fl.
MethodI-PD-R (↓)I-PD-R (↓)
full context26365.525864.0
Mem025462.726859.9
BM2524360.924459.9
LightMem10932.05214.9
StateMem (ours)29064.328649.1
Table 6: StateMemWrapper composition sweep; each backend ingests with its unmodified pipeline and is evaluated on the same frozen paired n=60 set per benchmark (StateMemBench: 30 short + 30 long scenarios, k=10; LongMemEval: k=20) under three answer conditions: the backend alone, a length- and cost-matched generic control (+Ctrl), and our state-tracing wrapper (+SMW). All answers are scored by a fixed deepseek-v4-pro judge. Best condition per row in bold. Dense on LongMemEval/Qwen has n=59 (one case failed at ingest).
Qwen-3.5-9Bdeepseek-v4-flash
BackendAlone+Ctrl+SMWAlone+Ctrl+SMW
StateMemBenchMem025.028.356.728.356.771.7
A-Mem23.335.056.733.351.775.0
LightMem3.330.061.71.751.768.3
MemOS5.025.056.71.750.068.3
BM2520.031.756.721.748.370.0
Dense21.728.356.731.743.370.0
LongMemEvalMem038.336.748.345.058.360.0
A-Mem41.731.756.735.051.755.0
LightMem5.011.728.33.331.735.0
MemOS6.78.323.36.731.736.7
BM2516.716.736.716.740.041.7
Dense47.532.250.853.361.756.7
Table 7: Inter-judge agreement on the binary drift decision: two same-model DeepSeek-V4 decodes (top), and DeepSeek-V4 × GPT-4o (bottom row cross family). MA-travel is omitted (2 doubly-labeled cases). MA-shopping is the one cell where κ misrepresents reliability: both passes call >90% of failures drift pre-filter, which crushes the chance correction despite 91.6% raw agreement.
Benchmarknagr (%)κPABAKAC1κmode
MA-shopping17991.60.170.830.910.00
MA-search11072.70.310.450.550.09
LME oracle3677.80.670.560.560.01
LME oracle (cross)3669.40.370.39
Table 8: Paired reasoning A/B. Reasoning does not close the gap: the −8.0 pp point estimate rests on 3 rescues against 7 new failures, which does not establish a direction (exact McNemar p=0.34). The off-pass → on-fail cell includes the case dissected in Appendix B.
Armcorrectrate
thinking off42/5084.0%
thinking on38/5076.0%
Δ−8.0 pp
Table 9: Source DOIs used to ground synthetic scenario generation, grouped by domain.
SourceDOI
Biomedical (bioRxiv)10.1101/482679
10.1101/512657
Computer Science (arXiv)10.48550/arXiv.2407.13248
10.48550/arXiv.2604.10985
10.48550/arXiv.2604.11543
10.48550/arXiv.2604.21253
10.48550/arXiv.2605.02392
10.48550/arXiv.2605.06196
Table 10: State-agreement accuracy on StateMemBench with deepseek-v4-pro judge. Left block: accuracy per horizon and n-weighted overall. Middle block: accuracy by probe category over all 322 probes (status n=116, sequence n=72, salience n=51, anti-trap n=39, compound n=44); Right block: accuracy by domain (finance n=93, shopping n=127, research n=102). Dense retrieval embeds with OpenAI text-embedding-3-small on both substrates; answers come from the panel’s substrate model. Per column within each panel, best in bold, second best underlined; StateMem ablations excluded from marking.
By horizonBy categoryBy domain
ConditionShortLongOverallStatusSequenceSalienceAnti-trapCompoundFinanceShoppingResearch
Long-contextQwen-3.5-9B0.1370.1670.1490.0780.0000.0980.6920.1590.1180.1970.118
Qwen-3.6-35B-A3B0.0630.1460.0970.0260.0000.0590.6320.0230.1100.1650.000
GPT-5.4-Nano0.3110.2270.2770.1720.1530.5690.6150.1140.3440.4020.059
DeepSeek-V4-Flash0.1320.1740.1490.1030.0560.0590.6410.0910.1610.2440.020
Qwen-3.5-9BNo memory0.0000.0150.0060.0090.0000.0000.0260.0000.0000.0000.020
BM250.0950.1670.1250.0340.0420.0000.8200.0230.1070.1020.167
Dense retrieval0.1050.1670.1300.0340.0140.0000.9230.0230.1180.1260.147
nano-graphrag0.1260.1670.1430.0350.0000.1000.8950.0680.1400.1490.137
HippoRAG0.1050.1670.1300.0340.0140.0000.7950.1360.1180.1260.147
LightRAG0.0890.1520.1150.0260.0140.0590.7690.0000.0650.1260.147
GraphRAG0.2160.2350.2240.1640.0690.1180.9230.1360.2040.1970.275
Mem00.1160.1970.1490.0260.0420.0390.8970.1140.1510.1260.177
A-Mem0.1000.1670.1270.0080.0420.0000.8970.0450.1180.1380.138
LightMem0.0210.0150.0190.0170.0000.0000.1020.0000.0000.0160.039
MemoryOS0.0260.0230.0250.0250.0000.0000.1020.0230.0220.0080.049
StateMem (ours)0.2370.2270.2330.1810.2080.0200.8460.1140.2690.2050.235
– extraction-only0.1260.1970.1550.0860.0000.0780.7950.1140.1610.1260.186
– + supersession0.1370.2200.1710.1120.0280.0390.8720.0910.1720.1650.176
– w/o dep.-prop.0.2020.2500.2220.1550.1390.0400.8970.1400.2150.2380.207
– w/o recompute0.1950.2270.2080.1900.0970.0590.8460.0450.2040.2130.206
deepseek-v4-flashNo memory0.0000.0080.0030.0000.0000.0000.0260.0000.0000.0000.010
BM250.1050.1970.1430.0340.0970.0000.8720.0230.1080.1100.216
Dense retrieval0.1680.2580.2050.0860.1810.0001.0000.0910.1940.2280.186
nano-graphrag0.1740.1970.1830.0950.0140.0980.9490.1140.2150.1730.167
HippoRAG0.1320.2580.1840.0860.1390.0000.9230.0680.1510.1730.225
LightRAG0.0580.1440.0930.0260.0280.0200.6150.0000.0650.1260.078
GraphRAG0.1370.2270.1740.1030.0420.1180.6920.1820.0970.1260.304
Mem00.1680.1890.1770.1030.0280.0980.9490.0230.1510.2050.167
A-Mem0.1580.2580.1990.0600.2640.0000.9490.0230.1940.1970.206
Table 11: Accuracy by question type on LongMemEval (top; full set, n=500, k=20) and LoCoMo (bottom; n=1,985; all retrieval k=20), both substrates (thinking off), graded by a fixed deepseek-v4-pro judge. Question-type n in the header. Knowledge-update and temporal-reasoning (LME) and temporal-reasoning (LoCoMo) are the drift-shaped subsets; LoCoMo adversarial rewards abstention. deepseek LME memory-system cells are from the final unified n=500 judge pass (all arms scored together). Per column within each panel, best in bold, second best underlined.
LongMemEval question type
ConditionOverallSS-user (70)SS-asst. (56)SS-pref. (30)Multi-sess. (133)Know.-upd. (78)Temporal (133)
Qwen-3.5-9BLong context0.5500.9000.9460.3330.4360.7440.248
No memory0.0620.0860.0540.0670.0530.0380.075
Mem00.5660.8710.6070.5330.6020.6030.338
A-Mem0.5200.8860.9290.4670.3610.6790.233
LightMem0.0640.0860.0730.0000.0530.0380.092
MemoryOS0.0800.1140.0360.1000.0600.1540.053
StateMem (ours)0.5800.8710.5360.7000.5190.7180.398
deepseek-flashLong context0.6660.9571.0000.6000.5940.7820.391
No memory0.0560.0860.0180.0000.0830.0380.053
Mem00.5940.9290.5890.3670.6170.6920.391
A-Mem0.5100.8860.8750.3330.3910.6790.218
LightMem0.0560.0860.0180.0000.0830.0380.053
MemoryOS0.0720.1140.0000.0000.0750.1790.030
StateMem (ours)0.6560.9430.6070.4000.5340.7950.624
Table 12: StateMemWrapper sweep on StateMemBench, by probe category and domain (gold %, frozen n=60 mixed set; category n: status 12, sequence 12, salience 13, anti-trap 13, compound 10; domain n: finance 22, shopping 18, research 20; deepseek-v4-pro judge). Anti-trap probes are controls (correct behaviour is not updating), hence high accuracy even for weak arms — and +SMW does not inflate there, i.e. it updates selectively rather than aggressively. Per-cell n is 10–13; splits are exploratory. +SMW rows shaded.
By categoryBy domain
BackendCond.StatusSeq.Sal.AntiComp.Fin.Shop.Res.
Qwen-3.5-9BMem0alone0809210271720
+Ctrl171723920233340
+SMW5058779240645670
A-Memalone00010010182820
+Ctrl8252310010272850
+SMW2550628560554470
LightMemalone000150000
+Ctrl833151000232260
+SMW4258468540505670
MemOSalone0002305010
+Ctrl0831770232230
+SMW4250549240555660
BM25alone000920181720
+Ctrl025318510232850
+SMW4258549230505070
Densealone0002300010
+Ctrl1725318510273950
+SMW3350468540505060
deepseek-v4-flashMem0alone17809220322820
+Ctrl1750698560645050
+SMW4267779280736790
A-Memalone2517010020363320
+Ctrl1733469270555050
+SMW4267928590866780
LightMemalone000800010
+Ctrl825467770554450
+SMW5067779280825680
MemOSalone00080500
+Ctrl2525548560555040
Table 13: StateMemWrapper sweep on LongMemEval: paired outcome transitions vs. the backend alone (frozen n=60; Dense/Qwen has n=59, one case failed at ingest. The set is drawn from the multi-session question type by construction, so no category split applies). For each backend, each condition’s answers are compared case-by-case against the alone answers: Fix = alone wrong → condition right; Brk = alone right → condition wrong; Fix − Brk equals the net case change in Table 6. +SMW is net-positive for every backend on both substrates; the control is net-negative for three Qwen backends (Mem0, A-Mem, Dense).
Qwen-3.5-9Bdeepseek-v4-flash
+Ctrl+SMW+Ctrl+SMW
BackendFix↑Brk↓Fix↑Brk↓Fix↑Brk↓Fix↑Brk↓
Mem010111812157156
A-Mem713178188164
LightMem40140170201
MemOS43100172202
BM2577153162172
Dense81712101271513
Table 14: Closed-pool failure modes on DeepSeek-V4-Flash, n=322 (combined Set A+B; errors are 0 for every method). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option (matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift / n).
MethodCorrectDriftOther-optOff-poolDrift-rate
Long-context (V4-Flash)4820646464.0%
Long-context (GPT5.4n)8915917349.4%
StateMem (ours)117158113649.1%
extraction-only5622663470.2%
+ supersession9618493357.1%
− dependency-propagation12015793648.8%
− recompute-guidance97179103655.6%
Mem057193185459.9%
A-Mem6418157256.2%
LightMem448027014.9%
MemoryOS883123025.8%
nano-graphrag5918876858.4%
HippoRAG59175127654.3%
LightRAG30150513746.6%
GraphRAG (MS)56158810049.1%
BM254619357859.9%
Dense66177136655.0%
No memory11803035.6%
Table 15: Closed-pool failure modes on Qwen-3.5-9B, n=322 (combined Set A+B; two cells n=320 after 2 grading drops, errors otherwise 0). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option (matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift / n).
MethodCorrectDriftOther-optOff-poolDrift-rate
Long-context-9B4821145965.5%
Long-context-35B3121566867.2%
StateMem (ours)7520783264.3%
extraction-only5023992474.2%
+ supersession55230102771.4%
− dependency-propagation71212132466.2%
− recompute-guidance6722072868.3%
Mem04820246862.7%
A-Mem4120647164.0%
LightMem6103021332.0%
MemoryOS8112120134.8%
nano-graphrag4619177859.3%
HippoRAG42194117560.2%
LightRAG371512011446.9%
GraphRAG (MS)7218546157.5%
BM254019677960.9%
Dense42209175464.9%
No memory278024224.2%

Why it matters

Agents that work across long, multi-session tasks can silently act on decisions or numbers that were already changed or cancelled, which is a real risk in domains like personal finance; this work gives a concrete way to measure and reduce that specific failure. It also gives practitioners a cheap wrapper they can bolt onto memory systems they already use, without redesigning the whole pipeline.

Terms in this paper

  • state drift · when the needed fact is present in context but the agent still acts on an outdated or overridden version of it
  • long-context baseline · feeding the entire conversation history directly into the model's input instead of using retrieval or a separate memory system
  • closed-pool grading · scoring an answer against a pre-defined small set of plausible options rather than free-form judging
  • retrieval-augmented (RAG) system · a setup where relevant text chunks are searched and pulled in before the model generates its answer
  • wrapper · a thin add-on that changes only the final answer-generation step of an existing system, without altering its internal machinery

Original abstract (English)

As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmarks focus largely on recall-shaped tasks, we argue an effective memory system must track the evolving state of the world; as facts, constraints, and decisions are revised over a long interaction, answers must reflect the current state and not a superseded one. We define this capability as state tracking and instantiate it in StateMemBench, a benchmark of 234 multi-session scenarios spanning two conversation-length regimes. Its closed-pool grading scores whether an answer reflects the current state, the superseded state, or fails otherwise, separating state-tracking failures from other errors by construction. Our analysis shows that this task is challenging for existing memory systems, retrieval-augmented baselines, and long-context baselines. We then present StateMem, a state-first memory method that explicitly tracks supersession and relational dependencies, and show it improves current-state accuracy over the strongest same-backbone baseline by 1.8x (0.205 -> 0.363) on DeepSeek-V4-Flash and over the strongest memory system by 1.6x (0.149 -> 0.233) on Qwen-3.5-9B, while remaining competitive with the long-context baselines. Finally, we show the same state approach can be applied as a lightweight single-call wrapper over existing memory systems, lifting current-state accuracy by +32 to +67 points on StateMemBench across six memory and retrieval backends. A length- and cost-matched control attributes +15 to +32 of those points to state structure rather than added context.

Authors · Xinyi Fan, Miri Liu, Ruozhen Yang, Siru Ouyang, Jiawei Han

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Xinyi Fan et al., arXiv:2608.19652, arxiv-nonexclusive