工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States

arXiv:2608.025082026-08-03

让自我进化的LLM智能体记忆越积越乱、奖励误发的问题,被压缩成四个固定格子解决

自我进化的LLM智能体会把过往经验存成记忆,并给每条记忆学习一个有用度分数,但随着经验累积,需要打分的对象不断增多,有限的反馈被稀释,而且一次任务成功的奖励会同时分给所有被共同检索出来的记忆,导致无关记忆也被错误地记上功劳,陷入所谓的记忆-奖励陷阱。RoMeRL不再让记忆空间无限增长,而是按结果极性(成功/失败)和记忆动态(长期代表/最近变化)交叉出四个固定格子来循环存放经验。在ALFWorld和LifelongAgentBench上的实验显示,性能提升的同时,冷启动比例、记忆规模和LLM调用次数都大幅下降。

METAL LAB 解读图

RoMeRL:把不断增长的记忆压缩进四个固定格子

证据状态已报告实测结果

  1. 原有问题每条存储轨迹都有单独的有用度分数,经验越多打分对象越多,共享的任务奖励又会错误地记到无关的共同检索记忆上
  2. 四个固定坐标按成功/失败与代表案例/最近变化交叉得到四格:PCC(最优成功案例)、PAC(失败后首次成功)、NCC(有价值的失败经验)、NAC(最近一次失败)
  3. 在线更新与替换新经验到来时只更新或替换对应格子的内容,并继承原有有用度分数作为热启动,而不新增永久变量
  4. 理论依据证明坐标越少每个坐标平均获得的反馈越多,并用污染-清理转移模型给出错误坐标残留比例的上界
  5. 实测效果在ALFWorld和LifelongAgentBench上:成功率提升,冷Q比例降低80.0%,反馈密度提升约6.0倍,记忆规模减少84.4%,LLM调用减少21.1%
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 问题所在:现有的可学习记忆系统给每条存储的轨迹单独分配一个有用度分数,经验越多,需要打分的对象空间就越大,有限的反馈被摊薄到越来越多的对象上。
  2. 记忆-奖励陷阱:一次任务的成功奖励会同时分给该次任务中被一起检索出来的所有记忆,导致实际上没有贡献的无关记忆也可能获得正向更新并持续存留。
  3. 解决方法:RoMeRL对每个任务只保留四个固定的语义坐标——按结果极性(成功/失败)与记忆动态(长期巩固代表/最近适应性变化)交叉得到PCC、PAC、NCC、NAC——新经验到来时只更新或替换这四格的内容,而不新增永久变量。
  4. 理论分析:作者证明减少坐标数量会提高每个坐标平均获得的反馈次数,并用一个通用的状态转移模型分析了错误坐标被污染又被清理的过程,给出错误坐标残留比例的上界。
  5. 实验结果:在ALFWorld和LifelongAgentBench(OS、DB任务)上,RoMeRL相比最强基线整体平均成功率提高3.2个百分点,从未获得更新的"冷Q"比例降低80.0%,反馈密度提升约6.0倍,记忆规模减少84.4%,LLM调用次数减少21.1%。
Figure 1: Memory-reward trap: Stronger exploration retrieves more low-access-count memories, including weakly relevant, noisy, outdated, or failed experiences. When such a memory appears in a successful episode, the shared trajectory reward can promote it despite no causal contribution. Exploration therefore increases feedback coverage but also raises the risk of reward misattribution.
Figure 1: Memory-reward trap: Stronger exploration retrieves more low-access-count memories, including weakly relevant, noisy, outdated, or failed experiences. When such a memory appears in a successful episode, the shared trajectory reward can promote it despite no causal contribution. Exploration therefore increases feedback coverage but also raises the risk of reward misattribution.
Table 1: Main results over 10 epochs. We compare RoMeRL with non-learning and learning-based agent-memory baselines. LAB reports Last-Epoch SR / CSR, while ALFWorld reports SR for six task types: P&P, examine, clean, heat, cool, and Pick-2. Overall is the macro-average of last-epoch SR across the two LAB tasks and six ALFWorld task types. For MemP, MemRL, and RoMeRL, we also report average LLM calls and memory-pool size.
MethodLifelong Agent BenchALFWorldOverall Avg.LLM Calls Avg Num.Memory Avg Num.
OS Last / CSROSLast / CSRDB Last / CSRDBLast / CSRP&PExamineCleanHeatCoolPick-2
OS
Last / CSR
DB
Last / CSR
ModelDS-V4-flashDS-V4-flashGPT-5.4-mini–––
No Memory0.6460.5500.8830.8270.8610.8500.8550.7880.783––
Pass@10– / 0.756– / 0.906–––––––––
RAG0.700 / 0.7520.556 / 0.8440.8910.8340.8680.8550.8580.7960.795––
Mem00.691 / 0.7330.575 / 0.8410.8970.8410.8730.8580.8720.8050.802––
MemP0.768 / 0.7960.631 / 0.942–––––––570K45K
MemRL0.808 / 0.8200.632 / 0.9340.9080.8550.8870.8650.8710.8120.830570K45K
RoMeRL (ours)0.824 / 0.8380.680 / 0.9520.9680.9570.9010.8620.8800.8260.862450K7K
Figure 2: Overview of RoMeRL. Top: the agent retrieves memories to construct the LLM context and updates memory utilities using task-level rewards. Although stronger exploration improves feedback coverage, it also expands the utility support exposed to erroneous reward propagation, increasing the risk of the memory-reward trap. Bottom: RoMeRL maps each task’s growing trajectory-indexed utility space into a fixed-dimensional state factorized by outcome polarity and memory dynamics. Its four semantic coordinates PCC, PAC, NCC, and NAC are updated through online retention, promotion, and replacement, thereby concentrating feedback over a bounded active support.
Figure 2: Overview of RoMeRL. Top: the agent retrieves memories to construct the LLM context and updates memory utilities using task-level rewards. Although stronger exploration improves feedback coverage, it also expands the utility support exposed to erroneous reward propagation, increasing the risk of the memory-reward trap. Bottom: RoMeRL maps each task’s growing trajectory-indexed utility space into a fixed-dimensional state factorized by outcome polarity and memory dynamics. Its four semantic coordinates PCC, PAC, NCC, and NAC are updated through online retention, promotion, and replacement, thereby concentrating feedback over a bounded active support.
Table 2: Controlled MRT stress test on the OS task. The first-round memory pool contains 10% noisy entries, and results are reported after ten training rounds. Positive Noise Updates denotes the average number of positive utility updates received by noisy entries over the ten rounds, whereas Final Noise Ratio denotes the percentage of noisy entries in the memory pool at the end of round 10.
MethodRound-10 SR (%) ↑Positive Noise Updates ↓Final Noise Ratio (%) ↓
MemRL79.23.71.02
MemRL + UCB78.47.21.20
RoMeRL82.02.40.15
Figure 3: Feedback density and Cold-Q ratio on the OS task. The Cold-Q ratio is the fraction of current representatives with no direct utility update since admission, i.e., nipost=0, while feedback density is their average number of post-admission updates.
Figure 3: Feedback density and Cold-Q ratio on the OS task. The Cold-Q ratio is the fraction of current representatives with no direct utility update since admission, i.e., nipost=0, while feedback density is their average number of post-admission updates.
Table 3: Cross-model memory transfer on LifelongAgentBench OS and DB tasks. Results with and without frozen transferred memory are reported as Validation Score / Average Steps (higher / lower is better), and Δ denotes the absolute score gain over the base agent.
Inference ModelBaseTransferGain (Δ)
LifelongAgentBench–OS
GPT-5.4-mini67.0 / 3.2381.6 / 2.22+14.6 / -1.01
Gemini-3.5-flash74.0 / 4.5381.4 / 3.02+7.4 / -1.51
LifelongAgentBench–DB
GPT-5.4-mini93.0 / 2.1596.8 / 2.00+3.8 / -0.15
Gemini-3.5-flash96.2 / 2.4497.6 / 2.18+1.4 / -0.26
Figure 4: Coordinate ablation on the OS task. Solid and dashed curves denote SR and CSR.
Figure 4: Coordinate ablation on the OS task. Solid and dashed curves denote SR and CSR.
Table 4: Model and API Configurations.
ComponentConfiguration / VersionNotes
Backbone LLMDS-V4-flashUsed for LifelongAgentBench
GPT-5.4-miniUsed for ALFWorld
Embedding ModelText-Embedding-3-LargeUsed for Intent and Query encoding
Generation ParamsTemperature =0.0General (Greedy decoding)
Top-p =1.0Default
Figure 5: Occupancy distribution of the factorized memory state on the OS and DB tasks. The bars report the percentage of occupied active-memory coordinates assigned to the PCC, PAC, NCC, and NAC.
Figure 5: Occupancy distribution of the factorized memory state on the OS and DB tasks. The bars report the percentage of occupied active-memory coordinates assigned to the PCC, PAC, NCC, and NAC.
Table 5: Hyperparameter Settings across Benchmarks.
Benchmark Setting
ParameterDescriptionLifelong Bench (OS)Lifelong Bench (DB)ALFWorld
RoMeRL (Ours)
αLearning Rate0.30.30.3
ωQQ-Weight Balance0.50.50.5
δSimilarity Threshold0.500.370.62
k1Cosine Similarity Recall Size10105
k2Final Memory Selection Size553
Qi​n​i​tInitial Q-value0.50.50.0
Baselines
kR​A​GRetrieval Top-k553
kS​e​l​f​R​A​GRetrieval Top-k553
kM​e​m​PRetrieval Top-k553
Figure 6: Final Q-value composition on OS tasks for the MemRL baseline (Left) and RoMeRL (Right). Each bar shows the proportions of success-derived and failure-derived memories within a Q-value interval; the legends report the fraction of memories remaining at the initial Q=0.5 and the Pearson correlation between Q-values and memory-generation outcomes.
Figure 6: Final Q-value composition on OS tasks for the MemRL baseline (Left) and RoMeRL (Right). Each bar shows the proportions of success-derived and failure-derived memories within a Q-value interval; the legends report the fraction of memories remaining at the initial Q=0.5 and the Pearson correlation between Q-values and memory-generation outcomes.
Table 6: Data split for Lifelong Agent benchmarks.
BenchmarkRuntime LearningTransfer LearningSplit / Note
Lifelong Agent (OS)500 tasks500 tasks7:3 Split (Seed 42)
Lifelong Agent (DB)500 tasks500 tasks7:3 Split (Seed 42)
RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States figure 6

研究结果

  • RoMeRL的整体平均成功率达到0.862,比最强基线的0.830高出3.2个百分点(Table 1)。
  • 在OS任务上,MemRL的冷Q比例从约29%升高到44.9%,而RoMeRL从约28%降到9.0%,反馈密度从4.96提升到29.93,约6.0倍(Figure 3)。
  • 平均记忆规模从45K降到7K,减少84.4%;LLM调用次数从570K降到450K,减少21.1%(Table 1)。
  • 在注入10%噪声的受控压力测试中,给MemRL加上UCB探索后,噪声条目获得的正向更新次数从3.7升到7.2,最终噪声比例从1.02%升到1.20%,而RoMeRL将两者分别限制在2.4和0.15%,同时取得最高的82.0%成功率(Table 2)。
  • 在四种模型-任务组合的跨模型记忆迁移实验中,迁移已学习的冻结记忆状态均提升了验证得分并减少了平均执行步数(Table 3)。
Figure 7: LLM calls per section on the DB and OS task. RoMeRL achieves lower and steadily decreasing call counts, while MemRL maintains a consistently high level.
Figure 7: LLM calls per section on the DB and OS task. RoMeRL achieves lower and steadily decreasing call counts, while MemRL maintains a consistently high level.

可应用场景

  • 为长期运行、经验不断累积的任务自动化或客服类智能体设计记忆管理结构
  • 在存储和LLM调用成本有限的智能体系统中控制资源开销
  • 将智能体切换到不同的LLM后端时,复用此前学到的过程性记忆
RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States figure 8

局限与待验证事项

  • 仍然依赖任务级的最终结果奖励,尚未完全解决记忆之间的因果功劳分配问题。
  • 要真正估计理论模型中的污染与清理转移参数(gamma、lambda),需要通过反事实回放等方式获得坐标级别的因果标签,这一点尚未验证。
  • 实验仅限于ALFWorld以及LifelongAgentBench的OS、DB子任务,更开放、更长视野的场景评估留待未来工作。
  • 四个固定坐标(2x2结构)是论文给出的最小设计,结合更细粒度奖励或其他划分标准的扩展尚未验证。

为什么重要

智能体运行时间越长,记忆越可能无限膨胀,带来存储成本和奖励误判的风险,这对长期运行的实际部署系统是个现实问题。这项工作说明,不需要重新训练底层LLM,仅通过改变记忆结构、把记忆压缩成固定大小,也能在保持或提升性能的同时大幅降低成本与污染风险。

本文术语

  • 冷Q比例(Cold-Q ratio) · 当前记忆代表项自加入以来从未获得直接有用度更新的比例,数值越低说明反馈越有效地传达到了记忆
  • 记忆-奖励陷阱(MRT) · 一次任务的共享奖励被同时记到所有共同检索出的记忆上,导致没有实际贡献的记忆也获得正向更新
  • PCC/PAC/NCC/NAC · RoMeRL维护的四个固定记忆格:正向巩固代表、正向适应性转变、负向巩固代表、负向最近失败
  • 累计成功率(CSR) · 在多个训练轮次(epoch)中至少被成功解决过一次的任务比例

论文原文摘要（英文）

Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL

作者 · Yi Yang

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Yi Yang et al., arXiv:2608.02508, arxiv-nonexclusive