Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents

arXiv:2608.195642026-08-21

AI assistants would rather double-check facts than ask you a question, even when asking is the right call

Researchers built MCB, a benchmark that tests whether an LLM agent correctly decides to permanently remember, use only once, re-verify, or ask the user about information it picks up during conversation. Testing Claude and Qwen models, they found both families were far more willing to verify facts against the world than to ask users to resolve ambiguity. Prompting tricks improved accuracy or reduced bad memory storage, but the failure to ask users when needed persisted across the board.

Original abstract (English)

Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification boundary: whether interaction-derived information should be persisted, used only in the current context, re-verified, or clarified with the user. MCB contains 140 primary scenarios, split into 70 development and 70 held-out items, plus a separate 70-item contrast set. It evaluates both action labels and structured tool-call selection. Two non-authors independently label the 70 held-out primary and 70 contrast items (97.1% agreement, Cohen's kappa = 0.962); a blind third resolves four disagreements, replacing eight author labels by non-author majority. Across Claude and Qwen, models verify changing facts more reliably than they ask users to resolve ambiguity. Bare Qwen asks on 0/12 clarification items while verifying 12/18 freshness items. Few-shot prompting raises accuracy from 0.557 to 0.771 (paired delta = +0.214, Holm-adjusted exact McNemar p_H = 0.002), yet clarification recall remains 0.333. The policy prompt reduces erroneous persistence from 0.243 to 0.100 (p_H = 0.038), although its accuracy gain is not significant. Label-tool agreement is 57% for each Claude model and 23% for Qwen; Qwen accuracy falls from 0.557 to 0.343 (p_H = 0.047). Memory evaluation must test both stated decisions and tool-call choices.

Authors · Baichuan Li, Junyi Yao, Zihao Zheng