Figure 1: StateMemBench targets state tracking: maintaining currently operative values of facts, rules, and derived quantities under cross-session revision, separable from recall.
Table 1: Cross-benchmark failure-mode distribution. Each cell is the share of that benchmark’s confirmed failures assigned to each bucket; the drift column is highlighted. †On LongMemEval oracle every gold fact is in the prompt by construction, so no failure there can be a retrieval failure. The small-n τ2-bench-Z result (n=16) is reported in §3.3. Labeling, adjudication, and per-row details: Appendix A.
Benchmark
n
drift
retr.
comp.
schema
reason
FA
MemoryArena
Shop.
200
63.5
22.5
2.5
0.0
9.5
–
Search
200
38.5
25.5
11.0
17.5
4.5
–
Travel
200
19.0
11.0
46.5
21.0
0.0
–
LongMemEval oracle
LME†
36
44.4
0.0
0.0
25.0
2.8
27.8
Figure 2: StateMem has three stages: an ingestion stage in which a conversation turn is parsed into state units, an update stage that updates the value of existing units, and a test time stage where the agent draws upon the valid state to answer the question. Here, we show how StateMem would handle a finance-based scenario.
Table 2: Comparison with agent-memory and long-horizon benchmarks with state-tracking evaluation. (✓) = partial satisfied. Scale is reported in each benchmark’s native unit. Column definitions and per-benchmark justifications are in Appendix C.
Benchmark
Multi-sess. dialogue
Scale (native unit)
Updates central
Super- session
Verifiable gold
Drift scored
Anti-update controls
Paired horizons
LoCoMo (20)
✓
∼300 turns
✗
✗
✗
✗
✗
✗
LongMemEval (30)
✓
∼40–80 sess. (S)
(✓)
(✓)
✗
✗
✗
✗
MemoryAgentBench (11)
(✓)
varies
(✓)
(✓)
(✓)
✗
✗
✗
MemoryArena (9)
(✓)
varies
✗
✗
✓
✗
✗
✗
STATE-Bench (21)
✗
450 tasks
✗
✗
(✓)
✗
(✓)
✗
StateMemBench (ours)
✓
∼600 turns
✓
✓
✓
✓
✓
✓
Table 3: State-agreement accuracy (gold rate) on StateMemBench: 190 in short scenarios (∼165 turns, one probe each) and 132 in 44 long fused scenarios (∼600 turns, three probes each), graded by a fixed deepseek-v4-pro judge. Memory systems (Mem0 (5), A-Mem (34), LightMem (7), MemoryOS (14), StateMem), retrieval baselines (BM25, text-embedding-3-small (22)), and graph-RAG systems (nano-graphrag, HippoRAG, LightRAG, GraphRAG) run on each substrate (Qwen-3.5-9B (25), thinking off; deepseek-v4-flash (33), thinking off); long-context baselines are raw models with full history in context. StateMem ablations (indented) excluded from best-marking. Per column within each panel, best bold, second best underlined. ***StateMem’s overall gold rate beats the strongest memory baseline on both backbones (paired McNemar test; both p < 0.001)
Condition
Overall
Short
Long
Long-context
Qwen-3.5-9B
0.149
0.137
0.167
Qwen-3.6-35B-A3B
0.097
0.063
0.146
GPT-5.4-Nano
0.277
0.311
0.227
DeepSeek-V4-Flash
0.149
0.132
0.174
Qwen-3.5-9B
No memory
0.006
0.000
0.015
BM25
0.125
0.095
0.167
Dense
0.130
0.105
0.167
nano-graphrag
0.143
0.126
0.167
HippoRAG
0.130
0.105
0.167
LightRAG
0.115
0.089
0.152
GraphRAG
0.224
0.216
0.235
Mem0
0.149
0.116
0.197
A-Mem
0.127
0.100
0.167
LightMem
0.019
0.021
0.015
MemoryOS
0.025
0.026
0.023
StateMem (ours)
0.233***
0.237
0.227
– extraction-only
0.155
0.126
0.197
– + supersession
0.171
0.137
0.220
– w/o dep.-propagation
0.222
0.202
0.250
– w/o recompute-guid.
0.208
0.195
0.227
deepseek-v4-flash
No memory
0.003
0.000
0.008
BM25
0.143
0.105
0.197
Dense
0.205
0.168
0.258
nano-graphrag
0.183
0.174
0.197
HippoRAG
0.184
0.132
0.258
LightRAG
0.093
0.058
0.144
GraphRAG
0.174
0.137
0.227
Mem0
0.177
0.168
0.189
A-Mem
0.199
0.158
0.258
LightMem
0.012
0.021
0.000
Table 4: Accuracy on LongMemEval (full set, n=500, k=20) and LoCoMo (n=1,985), with each method run on both substrates (Qwen-3.5-9B; deepseek-v4-flash; thinking off) and all answers graded by a fixed deepseek-v4-pro judge. Per column, best in bold, second best underlined.
LongMemEval
LoCoMo
Condition
Qwen-9B
DS-V4
Qwen-9B
DS-V4
Long context
0.550
0.666
0.612
0.587
No memory
0.062
0.056
0.010
0.005
Mem0
0.566
0.594
0.481
0.462
A-Mem
0.520
0.510
0.266
0.279
LightMem
0.064
0.056
0.027
0.020
MemoryOS
0.080
0.072
0.040
0.031
StateMem (ours)
0.580
0.656
0.566
0.592
Table 5: Closed-pool (n=322) drift-rate and in-pool engagement, methods that produce non-trivial in-pool counts on at least one backbone. In-pool (I-P) = method produced one of the closed-pool options. Drift-rate (D-R) = drift / n (errors are 0 for every method). StateMem’s I-P is bold.
Qwen-3.5-9B
DeepSeek-V4-Fl.
Method
I-P
D-R (↓)
I-P
D-R (↓)
full context
263
65.5
258
64.0
Mem0
254
62.7
268
59.9
BM25
243
60.9
244
59.9
LightMem
109
32.0
52
14.9
StateMem (ours)
290
64.3
286
49.1
Table 6: StateMemWrapper composition sweep; each backend ingests with its unmodified pipeline and is evaluated on the same frozen paired n=60 set per benchmark (StateMemBench: 30 short + 30 long scenarios, k=10; LongMemEval: k=20) under three answer conditions: the backend alone, a length- and cost-matched generic control (+Ctrl), and our state-tracing wrapper (+SMW). All answers are scored by a fixed deepseek-v4-pro judge. Best condition per row in bold. Dense on LongMemEval/Qwen has n=59 (one case failed at ingest).
Qwen-3.5-9B
deepseek-v4-flash
Backend
Alone
+Ctrl
+SMW
Alone
+Ctrl
+SMW
StateMemBench
Mem0
25.0
28.3
56.7
28.3
56.7
71.7
A-Mem
23.3
35.0
56.7
33.3
51.7
75.0
LightMem
3.3
30.0
61.7
1.7
51.7
68.3
MemOS
5.0
25.0
56.7
1.7
50.0
68.3
BM25
20.0
31.7
56.7
21.7
48.3
70.0
Dense
21.7
28.3
56.7
31.7
43.3
70.0
LongMemEval
Mem0
38.3
36.7
48.3
45.0
58.3
60.0
A-Mem
41.7
31.7
56.7
35.0
51.7
55.0
LightMem
5.0
11.7
28.3
3.3
31.7
35.0
MemOS
6.7
8.3
23.3
6.7
31.7
36.7
BM25
16.7
16.7
36.7
16.7
40.0
41.7
Dense
47.5
32.2
50.8
53.3
61.7
56.7
Table 7: Inter-judge agreement on the binary drift decision: two same-model DeepSeek-V4 decodes (top), and DeepSeek-V4 × GPT-4o (bottom row cross family). MA-travel is omitted (2 doubly-labeled cases). MA-shopping is the one cell where κ misrepresents reliability: both passes call >90% of failures drift pre-filter, which crushes the chance correction despite 91.6% raw agreement.
Benchmark
n
agr (%)
κ
PABAK
AC1
κmode
MA-shopping
179
91.6
0.17
0.83
0.91
0.00
MA-search
110
72.7
0.31
0.45
0.55
0.09
LME oracle
36
77.8
0.67
0.56
0.56
0.01
LME oracle (cross)
36
69.4
0.37
0.39
—
—
Table 8: Paired reasoning A/B. Reasoning does not close the gap: the −8.0 pp point estimate rests on 3 rescues against 7 new failures, which does not establish a direction (exact McNemar p=0.34). The off-pass → on-fail cell includes the case dissected in Appendix B.
Arm
correct
rate
thinking off
42/50
84.0%
thinking on
38/50
76.0%
Δ
−8.0 pp
Table 9: Source DOIs used to ground synthetic scenario generation, grouped by domain.
Source
DOI
Biomedical (bioRxiv)
10.1101/482679
10.1101/512657
Computer Science (arXiv)
10.48550/arXiv.2407.13248
10.48550/arXiv.2604.10985
10.48550/arXiv.2604.11543
10.48550/arXiv.2604.21253
10.48550/arXiv.2605.02392
10.48550/arXiv.2605.06196
Table 10: State-agreement accuracy on StateMemBench with deepseek-v4-pro judge. Left block: accuracy per horizon and n-weighted overall. Middle block: accuracy by probe category over all 322 probes (status n=116, sequence n=72, salience n=51, anti-trap n=39, compound n=44); Right block: accuracy by domain (finance n=93, shopping n=127, research n=102). Dense retrieval embeds with OpenAI text-embedding-3-small on both substrates; answers come from the panel’s substrate model. Per column within each panel, best in bold, second best underlined; StateMem ablations excluded from marking.
By horizon
By category
By domain
Condition
Short
Long
Overall
Status
Sequence
Salience
Anti-trap
Compound
Finance
Shopping
Research
Long-context
Qwen-3.5-9B
0.137
0.167
0.149
0.078
0.000
0.098
0.692
0.159
0.118
0.197
0.118
Qwen-3.6-35B-A3B
0.063
0.146
0.097
0.026
0.000
0.059
0.632
0.023
0.110
0.165
0.000
GPT-5.4-Nano
0.311
0.227
0.277
0.172
0.153
0.569
0.615
0.114
0.344
0.402
0.059
DeepSeek-V4-Flash
0.132
0.174
0.149
0.103
0.056
0.059
0.641
0.091
0.161
0.244
0.020
Qwen-3.5-9B
No memory
0.000
0.015
0.006
0.009
0.000
0.000
0.026
0.000
0.000
0.000
0.020
BM25
0.095
0.167
0.125
0.034
0.042
0.000
0.820
0.023
0.107
0.102
0.167
Dense retrieval
0.105
0.167
0.130
0.034
0.014
0.000
0.923
0.023
0.118
0.126
0.147
nano-graphrag
0.126
0.167
0.143
0.035
0.000
0.100
0.895
0.068
0.140
0.149
0.137
HippoRAG
0.105
0.167
0.130
0.034
0.014
0.000
0.795
0.136
0.118
0.126
0.147
LightRAG
0.089
0.152
0.115
0.026
0.014
0.059
0.769
0.000
0.065
0.126
0.147
GraphRAG
0.216
0.235
0.224
0.164
0.069
0.118
0.923
0.136
0.204
0.197
0.275
Mem0
0.116
0.197
0.149
0.026
0.042
0.039
0.897
0.114
0.151
0.126
0.177
A-Mem
0.100
0.167
0.127
0.008
0.042
0.000
0.897
0.045
0.118
0.138
0.138
LightMem
0.021
0.015
0.019
0.017
0.000
0.000
0.102
0.000
0.000
0.016
0.039
MemoryOS
0.026
0.023
0.025
0.025
0.000
0.000
0.102
0.023
0.022
0.008
0.049
StateMem (ours)
0.237
0.227
0.233
0.181
0.208
0.020
0.846
0.114
0.269
0.205
0.235
– extraction-only
0.126
0.197
0.155
0.086
0.000
0.078
0.795
0.114
0.161
0.126
0.186
– + supersession
0.137
0.220
0.171
0.112
0.028
0.039
0.872
0.091
0.172
0.165
0.176
– w/o dep.-prop.
0.202
0.250
0.222
0.155
0.139
0.040
0.897
0.140
0.215
0.238
0.207
– w/o recompute
0.195
0.227
0.208
0.190
0.097
0.059
0.846
0.045
0.204
0.213
0.206
deepseek-v4-flash
No memory
0.000
0.008
0.003
0.000
0.000
0.000
0.026
0.000
0.000
0.000
0.010
BM25
0.105
0.197
0.143
0.034
0.097
0.000
0.872
0.023
0.108
0.110
0.216
Dense retrieval
0.168
0.258
0.205
0.086
0.181
0.000
1.000
0.091
0.194
0.228
0.186
nano-graphrag
0.174
0.197
0.183
0.095
0.014
0.098
0.949
0.114
0.215
0.173
0.167
HippoRAG
0.132
0.258
0.184
0.086
0.139
0.000
0.923
0.068
0.151
0.173
0.225
LightRAG
0.058
0.144
0.093
0.026
0.028
0.020
0.615
0.000
0.065
0.126
0.078
GraphRAG
0.137
0.227
0.174
0.103
0.042
0.118
0.692
0.182
0.097
0.126
0.304
Mem0
0.168
0.189
0.177
0.103
0.028
0.098
0.949
0.023
0.151
0.205
0.167
A-Mem
0.158
0.258
0.199
0.060
0.264
0.000
0.949
0.023
0.194
0.197
0.206
Table 11: Accuracy by question type on LongMemEval (top; full set, n=500, k=20) and LoCoMo (bottom; n=1,985; all retrieval k=20), both substrates (thinking off), graded by a fixed deepseek-v4-pro judge. Question-type n in the header. Knowledge-update and temporal-reasoning (LME) and temporal-reasoning (LoCoMo) are the drift-shaped subsets; LoCoMo adversarial rewards abstention. deepseek LME memory-system cells are from the final unified n=500 judge pass (all arms scored together). Per column within each panel, best in bold, second best underlined.
LongMemEval question type
Condition
Overall
SS-user (70)
SS-asst. (56)
SS-pref. (30)
Multi-sess. (133)
Know.-upd. (78)
Temporal (133)
Qwen-3.5-9B
Long context
0.550
0.900
0.946
0.333
0.436
0.744
0.248
No memory
0.062
0.086
0.054
0.067
0.053
0.038
0.075
Mem0
0.566
0.871
0.607
0.533
0.602
0.603
0.338
A-Mem
0.520
0.886
0.929
0.467
0.361
0.679
0.233
LightMem
0.064
0.086
0.073
0.000
0.053
0.038
0.092
MemoryOS
0.080
0.114
0.036
0.100
0.060
0.154
0.053
StateMem (ours)
0.580
0.871
0.536
0.700
0.519
0.718
0.398
deepseek-flash
Long context
0.666
0.957
1.000
0.600
0.594
0.782
0.391
No memory
0.056
0.086
0.018
0.000
0.083
0.038
0.053
Mem0
0.594
0.929
0.589
0.367
0.617
0.692
0.391
A-Mem
0.510
0.886
0.875
0.333
0.391
0.679
0.218
LightMem
0.056
0.086
0.018
0.000
0.083
0.038
0.053
MemoryOS
0.072
0.114
0.000
0.000
0.075
0.179
0.030
StateMem (ours)
0.656
0.943
0.607
0.400
0.534
0.795
0.624
Table 12: StateMemWrapper sweep on StateMemBench, by probe category and domain (gold %, frozen n=60 mixed set; category n: status 12, sequence 12, salience 13, anti-trap 13, compound 10; domain n: finance 22, shopping 18, research 20; deepseek-v4-pro judge). Anti-trap probes are controls (correct behaviour is not updating), hence high accuracy even for weak arms — and +SMW does not inflate there, i.e. it updates selectively rather than aggressively. Per-cell n is 10–13; splits are exploratory. +SMW rows shaded.
By category
By domain
Backend
Cond.
Status
Seq.
Sal.
Anti
Comp.
Fin.
Shop.
Res.
Qwen-3.5-9B
Mem0
alone
0
8
0
92
10
27
17
20
+Ctrl
17
17
23
92
0
23
33
40
+SMW
50
58
77
92
40
64
56
70
A-Mem
alone
0
0
0
100
10
18
28
20
+Ctrl
8
25
23
100
10
27
28
50
+SMW
25
50
62
85
60
55
44
70
LightMem
alone
0
0
0
15
0
0
0
0
+Ctrl
8
33
15
100
0
23
22
60
+SMW
42
58
46
85
40
50
56
70
MemOS
alone
0
0
0
23
0
5
0
10
+Ctrl
0
8
31
77
0
23
22
30
+SMW
42
50
54
92
40
55
56
60
BM25
alone
0
0
0
92
0
18
17
20
+Ctrl
0
25
31
85
10
23
28
50
+SMW
42
58
54
92
30
50
50
70
Dense
alone
0
0
0
23
0
0
0
10
+Ctrl
17
25
31
85
10
27
39
50
+SMW
33
50
46
85
40
50
50
60
deepseek-v4-flash
Mem0
alone
17
8
0
92
20
32
28
20
+Ctrl
17
50
69
85
60
64
50
50
+SMW
42
67
77
92
80
73
67
90
A-Mem
alone
25
17
0
100
20
36
33
20
+Ctrl
17
33
46
92
70
55
50
50
+SMW
42
67
92
85
90
86
67
80
LightMem
alone
0
0
0
8
0
0
0
10
+Ctrl
8
25
46
77
70
55
44
50
+SMW
50
67
77
92
80
82
56
80
MemOS
alone
0
0
0
8
0
5
0
0
+Ctrl
25
25
54
85
60
55
50
40
Table 13: StateMemWrapper sweep on LongMemEval: paired outcome transitions vs. the backend alone (frozen n=60; Dense/Qwen has n=59, one case failed at ingest. The set is drawn from the multi-session question type by construction, so no category split applies). For each backend, each condition’s answers are compared case-by-case against the alone answers: Fix = alone wrong → condition right; Brk = alone right → condition wrong; Fix − Brk equals the net case change in Table 6. +SMW is net-positive for every backend on both substrates; the control is net-negative for three Qwen backends (Mem0, A-Mem, Dense).
Qwen-3.5-9B
deepseek-v4-flash
+Ctrl
+SMW
+Ctrl
+SMW
Backend
Fix↑
Brk↓
Fix↑
Brk↓
Fix↑
Brk↓
Fix↑
Brk↓
Mem0
10
11
18
12
15
7
15
6
A-Mem
7
13
17
8
18
8
16
4
LightMem
4
0
14
0
17
0
20
1
MemOS
4
3
10
0
17
2
20
2
BM25
7
7
15
3
16
2
17
2
Dense
8
17
12
10
12
7
15
13
Table 14: Closed-pool failure modes on DeepSeek-V4-Flash, n=322 (combined Set A+B; errors are 0 for every method). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option (matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift / n).
Method
Correct
Drift
Other-opt
Off-pool
Drift-rate
Long-context (V4-Flash)
48
206
4
64
64.0%
Long-context (GPT5.4n)
89
159
1
73
49.4%
StateMem (ours)
117
158
11
36
49.1%
extraction-only
56
226
6
34
70.2%
+ supersession
96
184
9
33
57.1%
− dependency-propagation
120
157
9
36
48.8%
− recompute-guidance
97
179
10
36
55.6%
Mem0
57
193
18
54
59.9%
A-Mem
64
181
5
72
56.2%
LightMem
4
48
0
270
14.9%
MemoryOS
8
83
1
230
25.8%
nano-graphrag
59
188
7
68
58.4%
HippoRAG
59
175
12
76
54.3%
LightRAG
30
150
5
137
46.6%
GraphRAG (MS)
56
158
8
100
49.1%
BM25
46
193
5
78
59.9%
Dense
66
177
13
66
55.0%
No memory
1
18
0
303
5.6%
Table 15: Closed-pool failure modes on Qwen-3.5-9B, n=322 (combined Set A+B; two cells n=320 after 2 grading drops, errors otherwise 0). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option (matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift / n).
As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmarks focus largely on recall-shaped tasks, we argue an effective memory system must track the evolving state of the world; as facts, constraints, and decisions are revised over a long interaction, answers must reflect the current state and not a superseded one. We define this capability as state tracking and instantiate it in StateMemBench, a benchmark of 234 multi-session scenarios spanning two conversation-length regimes. Its closed-pool grading scores whether an answer reflects the current state, the superseded state, or fails otherwise, separating state-tracking failures from other errors by construction. Our analysis shows that this task is challenging for existing memory systems, retrieval-augmented baselines, and long-context baselines. We then present StateMem, a state-first memory method that explicitly tracks supersession and relational dependencies, and show it improves current-state accuracy over the strongest same-backbone baseline by 1.8x (0.205 -> 0.363) on DeepSeek-V4-Flash and over the strongest memory system by 1.6x (0.149 -> 0.233) on Qwen-3.5-9B, while remaining competitive with the long-context baselines. Finally, we show the same state approach can be applied as a lightweight single-call wrapper over existing memory systems, lifting current-state accuracy by +32 to +67 points on StateMemBench across six memory and retrieval backends. A length- and cost-matched control attributes +15 to +32 of those points to state structure rather than added context.
