工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

arXiv:2608.084662026-08-08

不改动模型本身,而是让AI不断自己改写包裹模型的执行脚手架,结果性能提升了

LLM智能体的表现不仅取决于模型本身,还取决于hharness,也就是包裹在模型外面、由提示词、工具、记忆等组成的可执行脚手架代码。这篇论文提出了分层自我改进框架HSI,让一个参数被冻结、不再更新的LLM同时承担三个层级的角色,既改写具体任务的harness,也改写用来改写harness的策略本身。在BALROG基准测试上以DeepSeek-V4-Flash-Preview为冻结骨干模型进行实验,结果显示在中等难度任务上持续提升,但在超出模型自身能力的任务(NLE)上harness再怎么改也没有效果。

METAL LAB 解读图

不改动模型本身,而是让AI不断自己改写包裹模型的执行脚手架,结果性能提升了

  1. 01让同一个参数冻结的LLM分别承担三层角色:执行具体任务的harness、负责改写harness的进化者、以及负责改写进化者策略的元进化者,同时保持最外层执行逻辑不可修改,防止无限制的自我指涉
  2. 02在执行任务时关闭深度推理(thinking),只在自我修改阶段开启推理,从而把性能提升明确归因于harness改进本身,而非推理算力的增加
  3. 03在BALROG的中等难度任务上,相较初始harness,原始百分比进度分数提升了BabyAI +39.3、Crafter +33.0、TextWorld +25.0、MiniHack +15.0
  4. 04在BabaIsAI上,即便预留20%从未在改进过程中出现过的任务,进化后的harness在BreakStop上仍取得0.98、在GoTo上取得1.00的最佳测试成绩,说明确实具备泛化能力而非死记硬背
  5. 05而在超出模型能力上限的NLE任务上,无论怎样进化harness都没有带来实质提升,揭示了单靠改写harness存在明显天花板
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 让同一个参数冻结的LLM分别承担三层角色:执行具体任务的harness、负责改写harness的进化者、以及负责改写进化者策略的元进化者,同时保持最外层执行逻辑不可修改,防止无限制的自我指涉
  2. 在执行任务时关闭深度推理(thinking),只在自我修改阶段开启推理,从而把性能提升明确归因于harness改进本身,而非推理算力的增加
  3. 在BALROG的中等难度任务上,相较初始harness,原始百分比进度分数提升了BabyAI +39.3、Crafter +33.0、TextWorld +25.0、MiniHack +15.0
  4. 在BabaIsAI上,即便预留20%从未在改进过程中出现过的任务,进化后的harness在BreakStop上仍取得0.98、在GoTo上取得1.00的最佳测试成绩,说明确实具备泛化能力而非死记硬背
  5. 而在超出模型能力上限的NLE任务上,无论怎样进化harness都没有带来实质提升,揭示了单靠改写harness存在明显天花板
Figure 1: The HSI framework. A single frozen LLM M operates across three hierarchical scopes with disjoint editable surfaces. The task-harness scope executes the task-specific harness H; the evolver scope rewrites H through seed selection, main evolution, and commit selection; and the meta-evolver scope rewrites the evolver strategy Σ, including seed, evolution, commit, and final version selection policies.
Figure 1: The HSI framework. A single frozen LLM M operates across three hierarchical scopes with disjoint editable surfaces. The task-harness scope executes the task-specific harness H; the evolver scope rewrites H through seed selection, main evolution, and commit selection; and the meta-evolver scope rewrites the evolver strategy Σ, including seed, evolution, commit, and final version selection policies.
Table 1: BALROG leaderboard comparison under Setup A. The top block lists public-leaderboard numbers (retrieved 2026-08-03) from frontier models under their native configurations, reported as % Progress (15) (mean ± standard deviation across evaluation episodes). The bottom block reports HSI on the same task suite using a single frozen DeepSeek-V4-Flash backbone, isolating the contribution of the continuously evolvable hot-swappable task harness: the init-harness baseline, the meta-evolution-off arm, and the meta-evolution-on arm (bolded) that exports the deployed harness. BabaIsAI is omitted because our sub-suite protocol (§4.2.2) differs from the leaderboard’s mixed-task protocol. Avg reports the unweighted mean across the five environments, with its std the unweighted mean of the per-environment stds.
LLMBabyAICrafterTextWorldMiniHackNLEAvg
Gemini-3-Pro96.0±2.857.3±4.460.2±7.540.0±7.76.8±3.252.1±5.1
Gemini-3.1-Pro-Thinking98.0±2.055.0±6.475.7±6.427.5±7.12.6±0.351.8±4.4
Gemini-3.1-Pro100.0±0.046.8±4.266.5±7.535.0±7.53.0±0.550.3±3.9
Gemini-3-Flash86.0±4.945.0±6.350.2±8.130.0±7.24.0±0.843.0±5.5
Grok-476.0±6.057.3±3.962.9±7.917.5±6.01.8±0.843.1±4.9
Claude-Opus-4.580.0±5.749.5±3.151.4±8.427.5±7.12.0±0.542.1±5.0
Claude-Opus-4.5-Thinking72.0±6.348.6±3.259.0±8.030.0±7.22.4±0.342.4±5.0
Gemini-2.5-Pro-Exp-03-2580.0±5.755.0±6.049.2±8.217.5±6.01.7±0.240.7±5.2
DeepSeek-R174.0±6.236.4±3.821.8±6.125.0±6.81.4±0.531.7±4.7
GPT-5-minimal-think80.0±5.739.1±4.130.6±7.020.0±7.31.3±0.534.2±4.9
Claude-3.5-Sonnet68.0±6.632.7±3.242.1±5.415.0±5.60.6±0.531.7±4.3
GPT-4o77.6±3.733.1±2.339.3±5.210.0±4.70.4±0.432.1±3.3
DS-V4-Flash (Init harness)42.0±3.511.6±5.040.0±6.20.8±1.90.018.9±3.3
DS-V4-Flash w. HSI (meta-off)77.3±1.236.4±1.646.0±2.45.8±3.80.033.1±1.8
DS-V4-Flash w. HSI (meta-on)81.3±4.244.6±3.265.0±3.015.8±2.90.2±0.341.4±2.7
Figure 2: HSI evolution trajectory on Crafter (Setup A, meta-on). The dev reward climbs from the init-harness baseline 0.166 to an iteration-best 0.578 by iteration 4 (green-starred as the exported best version), with a regression in iteration 5. Each iteration’s annotation card reports four fields on the task-harness and evolver scopes (seed origin, main-evolution edit, result, commit pool) plus a green-tinted meta field summarizing what the meta-evolver rewrote in the evolver strategy Σ that iteration. The dominant lever uncovered by the evolver is making hidden game feedback explicit: reward signal, inventory state, and crafting feasibility, successively exposed in the harness context; the meta-evolver’s contribution is to codify these patterns into Σ so later iterations inherit them.
Figure 2: HSI evolution trajectory on Crafter (Setup A, meta-on). The dev reward climbs from the init-harness baseline 0.166 to an iteration-best 0.578 by iteration 4 (green-starred as the exported best version), with a regression in iteration 5. Each iteration’s annotation card reports four fields on the task-harness and evolver scopes (seed origin, main-evolution edit, result, commit pool) plus a green-tinted meta field summarizing what the meta-evolver rewrote in the evolver strategy Σ that iteration. The dominant lever uncovered by the evolver is making hidden game feedback explicit: reward signal, inventory state, and crafting feasibility, successively exposed in the harness context; the meta-evolver’s contribution is to codify these patterns into Σ so later iterations inherit them.
Table 2: BabaIsAI sub-suite results under Setup B (split evolution with 20% held-out test). “Best Dev” denotes the highest development reward selected during evolution. Test results are reported as mean ± across-task standard deviation of task progress. Init Harness is averaged over three baseline runs.
Sub-suiteInit HarnessBest DevBest Test (meta-on)Best Test (meta-off)
BreakStop0.0333±0.03341.00000.9800±0.06321.0000±0.0000
GoTo0.1818±0.08021.00001.0000±0.00000.9636±0.0809
Make0.00000.55560.3625±0.32840.3375±0.2029
Figure 3: HSI evolution trajectory on BabaIsAI-Make (Setup B, meta-on). The main blue curve is the dev reward across all reward_history evaluations; red rings mark commit-pool versions. Orange triangles (△) flag commits finalized in val mode (no dev evaluation recorded on the commit itself): each △ sits at the commit’s anchor x with the val reward on the y-axis, and a dashed vertical line drops from the dev anchor to the val reward so the dev→val evaluation structure reads directly off the trajectory. The dev peak climbs 0.222→0.333→0.444→0.556 across five iterations as the agent introduces a single-react rewrite with plan tracking (iter 1), a spatial-map builder with WIN-target persistence (iter 2), auto-target computation with BFS pathfinding (iter 3), an auto-push mechanism with directional fallback (iter 4), and LLM-aware cross-room navigation (iter 5); the meta-evolver codifies the “LLM targets, BFS navigates” two-layer pattern and progressively expands the commit pool from one to three versions. The △ markers expose which iterations’ primary commits were validated on held-out val tasks, an evaluation structure unique to Setup B.
Figure 3: HSI evolution trajectory on BabaIsAI-Make (Setup B, meta-on). The main blue curve is the dev reward across all reward_history evaluations; red rings mark commit-pool versions. Orange triangles (△) flag commits finalized in val mode (no dev evaluation recorded on the commit itself): each △ sits at the commit’s anchor x with the val reward on the y-axis, and a dashed vertical line drops from the dev anchor to the val reward so the dev→val evaluation structure reads directly off the trajectory. The dev peak climbs 0.222→0.333→0.444→0.556 across five iterations as the agent introduces a single-react rewrite with plan tracking (iter 1), a spatial-map builder with WIN-target persistence (iter 2), auto-target computation with BFS pathfinding (iter 3), an auto-push mechanism with directional fallback (iter 4), and LLM-aware cross-room navigation (iter 5); the meta-evolver codifies the “LLM targets, BFS navigates” two-layer pattern and progressively expands the commit pool from one to three versions. The △ markers expose which iterations’ primary commits were validated on held-out val tasks, an evaluation structure unique to Setup B.
Table 3: Per-suite experimental configuration. Shared settings are listed in the prose above; only the entries below vary across suites.
SuiteSetupDevValTest ep.Dev ep.Test rep.MetaSubmit-best
TextWorldA1.00.001033off50
BabyAIA1.00.001033on80
CrafterA1.00.00533on50
MiniHackA1.00.00513on80
NLEA1.00.00511on50
BabaIsAI-BreakStopB0.80.20511off80
BabaIsAI-GoToB0.80.25511on80
BabaIsAI-MakeB0.80.25511on80
Table 4: Representative harness evolution and self-improvement methods compared with HSI. Proposer: who proposes harness edits; Surface: the editable code surface; Domain: primary evaluation domain; Feature: the method’s distinguishing contribution in short form.
MethodProposerSurfaceDomainFeature
Meta-HarnessExternal strongerFull harness codeCoding, mathFull-trajectory feedback
Self-HarnessSelf (target)Config interfaceCodingModel-specific edits
AHEExternal strongerDecoupled compsCodingObservability bottleneck
HarnessXMulti-agentTyped processors5 benchmarksOperational mirror to RL
DGMSelfFull codebaseCodingArchive-based search
HGMSelfFull codebaseCodingClade meta-productivity
GEASelf (group)Codebase + expCodingShared experience pool
SICASelfFull codebaseCodingFramework saturation
HyperAgentsFused selfCodebase + metaCoding, robotsEditable meta-mechanism
Live-SWE-AgentSelf (runtime)Tools on-the-flyCodingZero offline cost
TTHESelf (test-time)Harness populationCoding, SQLUnlabeled trace adaptation
Rethinking Eval.— (critique)—CodingTest-time-scaling confound
Statistical Limits— (theory)—PAC learningVC bound ⇔ learnability
HSI (Ours)Same frozen M3-layer hierarchyBALROGEndogenous hierarchy with frozen outer anchor

为什么重要

以往提升智能体大多依赖人工手动调整提示词或工具,这项研究证明外围的执行脚手架本身也可以由AI分层、安全地自我改写,而无需更新模型权重。同时它也用实证方式划出了这种自我改进失效的边界,为开发者判断该投入改进harness还是升级底层模型提供了依据。

本文术语

  • harness · 包裹在语言模型外部、由提示词、工具、记忆、控制逻辑组成的可执行脚手架代码
  • 冻结的LLM(frozen LLM) · 训练好的参数保持固定、不再更新的语言模型
  • 进化者/元进化者(evolver/meta-evolver) · 分别负责改写具体任务harness、以及改写改写策略本身的角色
  • % Progress · BALROG基准中用0到100分表示任务完成程度的指标
  • 留出集泛化(held-out generalization) · 用改进过程中完全未接触过的任务来测试,以确认效果能否迁移到新场景

论文原文摘要（英文）

Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the harness---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is task-specific and continuously evolvable: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce Hierarchical Self-Improvement (HSI), a framework in which a single frozen LLM M operates across three hierarchical scopes: a task harness H that executes tasks, an evolver that rewrites H, and a meta-evolver that rewrites the evolver's strategy code under a frozen outer anchor. A thinking-on/off design isolates the contribution of harness evolution by disabling reasoning during task execution while enabling it during self-modification. HSI is bounded by two factors: a feedback-fidelity bound, since evolution requires informative reward signals to guide selection, and a backbone capability bound, since harness redesign cannot overcome limitations of the frozen model. On BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI achieves consistent gains over the initial harness on moderate-difficulty tasks (+39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack, all in raw \% Progress), while obtaining strong held-out generalization on BabaIsAI sub-suites (0.98 best-test on BreakStop and 1.00 on GoTo from a 20% unseen split). On tasks beyond the backbone's capability (NLE), harness evolution provides no improvement. These results demonstrate task-specific harness evolution as a viable axis for improving frozen LLM agents under clear empirical limits. Code is available at https://github.com/TailinZhou/hsi.

作者 · Tailin Zhou

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Tailin Zhou et al., arXiv:2608.08466, CC BY 4.0