Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration
arXiv:2608.197012026-08-21
When multiple AI agents' memories are secretly copies of the same source, this method stops the system from being fooled by a fake majority
In long-running multi-agent AI systems, different agents write to a shared memory store, but memories written by different agents can secretly trace back to the same original source. Existing systems count these copies as separate independent votes, creating a false majority the researchers call Memory Correlation Bias. The proposed CAMA framework detects hidden overlaps between memories, estimates how much truly independent evidence exists, and actively fetches more evidence when needed, outperforming state-of-the-art baselines across multiple benchmarks.
What they did
Problem: when systems simply vote or weight retrieved memories from multiple agents, memories that are actually duplicates of the same upstream source get counted repeatedly as if independent, inflating support for a wrong answer -- a failure the authors name Memory Correlation Bias
Method: CAMA uses a neural module to infer how much retrieved memories overlap in the evidence they carry, combined with provenance (source-tracking) information, to estimate the effective number of independent evidence sources; when evidence is insufficient it learns a policy to either retrieve more memories or trace memories back to their upstream origin
Results: tested on three benchmarks (MemoryAgentBench, LongMemEval, LoCoMo) with two backbone models (DeepSeek-V4-Flash and Qwen3.6-27B), CAMA consistently beat baselines including Vanilla RAG, Majority Voting, Mem0, HippoRAG, and multi-agent methods MAD and MADAM-RAG, recovering correct answers more often even when they were outnumbered by correlated duplicates, and staying more stable as redundant memories increased
Efficiency: compared to multi-agent baselines, CAMA needed fewer LLM calls and less token usage while achieving a better accuracy gain per token, showing the gains come from structured evidence reasoning rather than repeated agent interactions
Figure 1: An overview of our proposed CAMA. The diagram illustrates the overall workflow of memory arbitration, where retrieved memories are progressively processed through evidence decoupling, conflict arbitration, and evidence recovery.
Table 1: Overall performance comparison on three benchmark datasets. Best results are marked by bold.
Backbone
Methods
MemoryAgentBench
LongMemEval
LOCOMO
FC-SH
FC-MH
Overall
EM
F1
BERT
Judge
EM
F1
BERT
Judge
DeepSeek-V4-Flash
Vanilla RAG
68.4
34.2
51.3
34.1
45.7
84.2
51.6
29.7
40.3
83.5
47.2
Majority Voting
70.1
33.5
51.8
33.6
45.1
84.0
50.9
28.9
39.6
83.2
46.1
HippoRAG
72.6
42.8
57.7
38.5
49.8
85.3
56.4
33.4
44.1
84.6
51.8
Mem0
73.9
44.5
59.2
40.2
51.6
85.7
58.1
35.8
46.7
85.2
54.3
MAD
74.7
46.9
60.8
41.7
52.9
86.1
60.5
36.9
47.8
85.5
56.2
MADAM-RAG
75.2
48.8
63.4
44.1
54.3
86.8
62.4
37.6
49.5
85.7
59.4
CAMA (Ours)
78.9
55.7
67.3
49.8
59.1
87.9
69.2
43.6
53.8
87.1
64.7
Qwen3.6-27B
Vanilla RAG
65.2
31.4
48.3
31.5
42.9
83.4
48.7
27.3
37.8
82.7
44.5
Majority Voting
66.8
30.7
48.8
30.9
42.3
83.1
47.9
26.5
37.1
82.4
43.6
HippoRAG
69.5
39.6
54.6
35.6
46.8
84.5
53.2
30.9
41.4
83.8
49.1
Mem0
71.0
41.3
56.2
37.4
48.7
84.9
55.3
33.2
43.9
84.4
51.7
MAD
71.9
43.8
57.9
38.9
50.1
85.3
57.6
34.5
45.2
84.7
53.8
MADAM-RAG
73.6
46.7
60.2
41.6
52.6
85.9
61.2
36.8
47.5
85.3
57.1
CAMA (Ours)
76.5
53.2
64.9
47.4
56.9
87.2
67.0
41.5
51.6
86.5
62.4
Figure 2: Hyperparameter sensitivity analysis on MemoryAgentBench with DeepSeek-V4-Flash.
Table 2: Evaluation of memory correlation bias mitigation under DeepSeek-V4-Flash.
Methods
MemoryAgentBench
LongMemEval
LOCOMO
CMR ↑
RS ↓
IEG ↑
ERR ↑
CMR ↑
RS ↓
IEG ↑
ERR ↑
CMR ↑
RS ↓
IEG ↑
ERR ↑
Vanilla RAG
38.7
41.2
5.8
5.3
36.9
43.5
5.1
4.6
33.4
45.8
4.4
3.9
Majority Voting
33.5
44.8
4.7
4.9
31.8
46.9
4.2
4.2
28.7
49.3
3.5
3.5
HippoRAG
46.8
27.4
11.6
9.8
44.2
29.1
10.5
8.9
40.5
31.6
9.1
7.7
Mem0
49.6
24.1
13.7
11.2
47.1
25.8
12.6
10.3
43.4
28.2
11.0
9.0
MAD
54.3
19.2
15.9
12.6
51.8
20.7
14.5
11.4
47.6
22.9
12.8
10.1
MADAM-RAG
60.7
15.3
19.4
14.1
58.2
16.6
18.1
12.9
53.9
18.5
16.2
11.5
CAMA (Ours)
71.2
7.8
25.1
36.2
67.4
9.1
22.6
33.1
62.1
10.3
20.2
29.4
(b)
Table 3: Ablation study on the MemoryAgentBench and LongMemEval benchmarks under DeepSeek-V4-Flash.
Methods
MemoryAgentBench
LongMemEval
FC-SH
FC-MH
Overall
CMR
RS
IEG
ERR
EM
F1
BERT
Judge
CMR
RS
IEG
ERR
w/o Evi. Decoupling
71.5
45.3
58.4
48.2
32.7
14.6
28.9
42.1
51.8
86.2
59.7
46.5
34.1
13.8
26.4
w/o Prov. Prior
76.8
51.9
64.4
63.4
13.8
21.7
34.2
47.3
56.6
87.4
65.8
61.2
14.6
20.3
31.8
w/o Expand
76.2
51.4
63.8
66.7
9.5
17.2
15.3
46.1
55.7
87.3
64.9
63.8
10.2
15.8
13.6
w/o Trace
75.4
49.6
62.5
64.8
15.2
19.4
27.1
46.8
56.2
87.4
65.7
62.5
16.3
18.1
25.2
w/o Policy
77.1
52.3
64.7
65.9
11.4
16.8
22.7
47.6
57.0
87.5
66.4
64.1
12.5
15.2
20.8
CAMA
78.9
55.7
67.3
71.2
7.8
25.1
36.2
49.8
59.1
87.9
69.2
67.4
9.1
22.6
33.1
(c)
Table 4: Efficiency analysis on the MemoryAgentBench benchmark under DeepSeek-V4-Flash.
Methods
Avg. Latency
LLM Calls
Token Cost (k)
Context Len (k)
Δ Acc./ kToken
Vanilla RAG
1.8
1.0
3.2
3.1
–
Majority Voting
4.6
5.0
12.8
3.1
0.04
HippoRAG
3.2
2.0
5.9
4.4
1.08
MAD
9.8
8.4
24.3
9.6
0.39
MADAM-RAG
11.4
10.6
28.9
11.2
0.42
CAMA(Ours)
6.7
4.2
14.6
6.8
1.10
(d)
Table 5: Evaluation of memory correlation bias mitigation under Qwen3.6-27B.
Methods
MemoryAgentBench
LongMemEval
LOCOMO
CMR ↑
RS ↓
IEG ↑
ERR ↑
CMR ↑
RS ↓
IEG ↑
ERR ↑
CMR ↑
RS ↓
IEG ↑
ERR ↑
Vanilla RAG
36.2
43.1
5.1
4.7
34.5
45.4
4.6
4.1
31.1
47.5
4.0
3.5
Majority Voting
31.4
46.5
4.1
4.3
29.8
48.6
3.7
3.8
26.8
51.0
3.1
3.1
HippoRAG
43.9
29.1
10.4
8.7
41.5
30.8
9.5
7.9
38.0
33.4
8.2
6.9
Mem0
46.7
25.7
12.3
10.1
44.4
27.4
11.4
9.3
40.8
29.8
9.9
8.2
MAD
51.2
20.6
14.5
11.5
48.9
22.1
13.2
10.4
44.8
24.4
11.7
9.2
MADAM-RAG
57.5
16.7
17.7
13.0
55.0
18.1
16.5
11.8
50.8
20.1
14.8
10.5
CAMA (Ours)
68.1
8.6
23.4
34.0
64.3
10.0
21.1
31.0
59.0
11.2
18.8
27.5
Table 6: Ablation study on the MemoryAgentBench and LongMemEval benchmarks under Qwen3.6-27B.
Methods
MemoryAgentBench
LongMemEval
FC-SH
FC-MH
Overall
CMR
RS
IEG
ERR
EM
F1
BERT
Judge
CMR
RS
IEG
ERR
w/o Evi. Decoupling
69.3
43.5
56.4
46.0
34.1
13.2
27.0
40.3
50.1
85.6
57.8
44.7
35.4
12.5
24.8
w/o Prov. Prior
74.6
49.8
62.2
60.8
14.9
20.2
32.1
45.7
55.0
86.8
63.9
58.6
15.7
18.9
29.8
w/o Expand
74.1
49.3
61.7
63.8
10.6
15.9
14.2
44.6
54.1
86.7
63.0
60.9
11.3
14.7
12.5
w/o Trace
73.4
47.6
60.5
61.9
16.2
18.0
25.4
45.2
54.7
86.8
63.8
59.7
17.4
17.0
23.7
w/o Policy
75.0
50.5
62.8
63.1
12.4
15.6
21.2
46.1
55.6
86.9
64.5
61.4
13.6
14.2
19.6
CAMA
76.5
53.2
64.9
68.1
8.6
23.4
34.0
47.4
56.9
87.2
67.0
64.3
10.0
21.1
31.0
Table 7: Efficiency analysis on the MemoryAgentBench benchmark under Qwen3.6-27B.
Methods
Avg. Latency
LLM Calls
Token Cost (k)
Context Len (k)
Δ Acc./ kToken
Vanilla RAG
1.5
1.0
3.2
3.1
–
Majority Voting
3.9
5.0
12.8
3.1
0.04
HippoRAG
2.8
2.0
5.9
4.4
1.07
MAD
8.3
8.4
24.3
9.6
0.40
MADAM-RAG
9.7
10.6
28.9
11.2
0.41
CAMA(Ours)
5.8
4.2
14.6
6.8
1.14
Why it matters
In systems where AI agents keep writing decisions back into shared memory over long periods, a single duplicated error can snowball into a persistent, self-reinforcing mistake -- this work offers a concrete way to catch that before it compounds. It matters for anyone building or relying on long-term multi-agent AI systems that need trustworthy, non-repetitive evidence for their decisions.
Terms in this paper
Memory Correlation Bias · When memories from different agents actually share the same origin but get counted as independent evidence, creating a false majority
provenance · Recorded information about where and how a memory entry was originally created or derived
CAMA (Correlation-Aware Memory Arbitration) · The proposed method that detects hidden dependencies between memories, estimates independent evidence, and recovers missing evidence before deciding
effective independent evidence (Neff) · A calculated measure of how many genuinely distinct pieces of evidence exist among the retrieved memories
recovery policy · A learned decision-making process that chooses to retrieve more memories, trace a memory's origin, or stop, when current evidence is insufficient
Original abstract (English)
Long-term multi-agent systems continuously accumulate the memories produced by different agents. Existing memory methods typically treat retrieved memories as independent evidence and combine them through voting or weighting. However, this independence assumption often fails in multi-agent settings: memories written by different agents may inherit the same upstream source or shared bias, causing correlated evidence to be repeatedly counted and creating a false majority. We term this failure mode \textit{Memory Correlation Bias}. To address the issue, we propose the \textbf{C}orrelation-\textbf{A}ware \textbf{M}emory \textbf{A}rbitration (CAMA) framework that jointly decouples retrieved memories and recovers missing independent evidence. We model the retrieved memories as query-conditioned evidence groups and combine neural dependency inference with provenance-based symbolic priors to estimate the effective number of independent evidence sources, thereby preventing correlated memories from forming a false majority. Since critical independent evidence may be absent from the initial retrieval set, \textsc{CAMA} further learns a sequential recovery policy that actively retrieves alternative evidence or traces upstream sources before making the final decision, aiming to recover sufficient independent evidence for reliable arbitration while minimizing retrieval cost. Experiments on multiple benchmarks demonstrate the superiority of our method over the state-of-the-art baseline methods, suppressing false majorities induced by correlated memories.
