每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

EnvHarness: Awakening Static Worlds for Agent Learning

arXiv:2608.198802026-08-21

不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形

LLM智能体靠与环境互动来学习,但人工搭建的环境是静止不变的,既发现不了智能体的具体弱点,也会在智能体进步后很快变得毫无挑战。EnvHarness在不改动环境本身逻辑和其可信的验证器的前提下,给环境套上小型可插拔组件——改变初始状态(Stage)、改变允许的动作与观察内容(Contract)、或把多个任务串联成更长的一个任务(Chain)。配套的EnvRigger系统会观察智能体的成功与失败轨迹,自动编写针对其弱点的组件,并通过重新试跑来验证有效后才采纳。

他们做了什么

  1. 问题所在:现有的智能体训练环境是人工搭建且固定不变的,无法针对某个智能体的具体弱点,而且一旦智能体学会解决现有任务,环境就没有更多可教的东西了;此前的自动环境生成方法又局限于特定领域,且依赖不可靠的LLM生成验证器。
  2. 解决方案:借鉴给冻结的大模型加装工具、记忆等外部组件使其变成智能体的思路(agent harness),把同样的原理用在环境一侧——保持原环境的运行逻辑和验证器完全不变,套上三类组件:Stage(改变起始状态)、Contract(改变允许的动作/观察/反馈)、Chain(把两个环境串联成一个更长的任务)。
  3. 自动化机制:EnvRigger把目标智能体当作黑箱,只观察其执行轨迹,诊断出具体问题(比如重复动作循环、读不懂长文本、钻空子走捷径),据此编写候选组件,再用新一轮试跑验证效果,只有验证通过的组件才会被采纳,否则修改或放弃。
  4. 实验结果:在ALFWorld、WebArena、SWE-bench Verified、OfficeQA、SpreadsheetBench共四个领域五个基准上测试,用EnvHarness改造后的环境训练出的智能体,表现优于用原始环境训练的智能体,也优于各领域专用的环境生成方法,在留出测试任务上最高提升9.0分,同时在SWE-bench Verified上执行步数减少9.8%。
  5. 额外发现:同样的方法也能为强化学习提供更好的训练信号;随着环境数量增加,EnvHarness效果持续提升,而原始环境和生成环境的效果会趋于平缓;该方法在从弱到强的不同模型骨干上都表现一致。
EnvHarness: Awakening Static Worlds for Agent Learning figure 0
EnvHarness: Awakening Static Worlds for Agent Learning figure 1
EnvHarness: Awakening Static Worlds for Agent Learning figure 2
Table 7 | Training and evaluation splits per benchmark.
BenchmarkTrainingEvaluation
ALFWorld100 tasks from the standard train setall remaining held-out tasks
WebArena20 tasks per sub-domainall remaining tasks
SWE-bench100 tasks from SWE-bench Lite407 Verified issues not in Lite
OfficeQA50 tasks (official split)172 official test tasks
SpreadsheetBench100 of the 400 verified tasks299 held-out tasks (897 instances)
Figure 2 | While an agent harness transforms a frozen LLM into a capable agent via plug-in components (e.g., skills, memory, tools) without altering model weights, EnvHarness applies this same principle to the other side of the interaction. It customizes a frozen environment with plug-in components while leaving original environment unchanged.
Figure 2 | While an agent harness transforms a frozen LLM into a capable agent via plug-in components (e.g., skills, memory, tools) without altering model weights, EnvHarness applies this same principle to the other side of the interaction. It customizes a frozen environment with plug-in components while leaving original environment unchanged.
Figure 3 | Overview of EnvHarness components wrapping the standard environment inter- face. The underlying base environment (native state transitions and original task verifier) remains completely frozen. From left to right: the base environment, followed by three EnvHarness compo- nents—Stage, Contract, and Chain. Highlighted arrows and headers indicate overridden interface methods, with code blo
Figure 3 | Overview of EnvHarness components wrapping the standard environment inter- face. The underlying base environment (native state transitions and original task verifier) remains completely frozen. From left to right: the base environment, followed by three EnvHarness compo- nents—Stage, Contract, and Chain. Highlighted arrows and headers indicate overridden interface methods, with code blo
EnvHarness: Awakening Static Worlds for Agent Learning figure 5

为什么重要

对于训练AI智能体的人来说,这提供了一种无需为每个新版本智能体重新设计训练环境、就能让环境持续保持有效的方法。先诊断智能体的具体弱点再针对性改造练习环境的思路,有望通过同一套接口应用到编程、网页操作、办公自动化、具身任务等多个领域。

EnvHarness: Awakening Static Worlds for Agent Learning figure 6
EnvHarness: Awakening Static Worlds for Agent Learning figure 7
Figure 4 | EnvRigger generating EnvHarness components for a target policy based on given task. The execution loop on the left runs the policy against the current environment, which is a frozen base environment wrapped by the active EnvHarness containing accepted components 𝑤1, . . . , 𝑤𝑘, while the resulting rollout trajectories feed the EnvRigger loop on the right. The EnvRigger operates syste
Figure 4 | EnvRigger generating EnvHarness components for a target policy based on given task. The execution loop on the left runs the policy against the current environment, which is a frozen base environment wrapped by the active EnvHarness containing accepted components 𝑤1, . . . , 𝑤𝑘, while the resulting rollout trajectories feed the EnvRigger loop on the right. The EnvRigger operates syste

本文术语

  • LLM智能体 · 基于大语言模型构建、能在环境中采取实际行动(如点击网页、修改代码)而不仅仅是回答问题的AI系统
  • 环境与验证器 · 智能体所互动的模拟任务场景,以及用来判断任务是否成功完成的评分机制
  • Stage / Contract / Chain · EnvHarness的三种组件类型:Stage改变初始状态,Contract改变允许的动作/观察内容,Chain把两个任务串联成一个更长的任务
  • 黑箱观察 · 不查看模型内部权重,只根据输入输出和行为轨迹来分析智能体
  • 强化学习(RL)/ GRPO · 通过试错获得的奖励信号来改进策略的训练方法,GRPO是本文使用的一种具体优化算法

论文原文摘要（英文）

LLM agents learn by interacting with environments, yet these environments are hand-built and static: blind to an agent's weaknesses, and quickly left behind as it improves. While recent environment generation methods attempt to address this, they require domain-specific pipelines, rely on expensive or unreliable verifiers, and still produce static environments. To alleviate the engineering burden of rebuilding environments from scratch, we propose Environment Harness (EnvHarness), a programmable layer of plug-in components that wraps a static environment to reshape its behavior without modifying the underlying logic. Operating through standard interfaces, EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier. To automate this process, we introduce EnvRigger, which treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws, and validating them via fresh rollouts. Across five benchmarks in four domains, EnvHarness outperforms both original environments and domain-specific environment generation pipelines, achieving up to a 9.0-point improvement on held-out instances with 9.8% fewer execution steps. Furthermore, EnvHarness provides a superior optimization signal for reinforcement learning, enabling continuous, targeted co-evolution of the policy and its environment.

作者 · Chengsong Huang, Zifeng Wang, Rujun Han, Jun Yan, Yanfei Chen, Zoey CuiZhu, Ke Jiang, Peng Xia, Han Yu, Yufan Zhuang, Yi

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Chengsong Huang et al., arXiv:2608.19880, CC BY 4.0