DarwinX: Evolving Agent Harnesses Through Natural Selection

arXiv:2608.075452026-07-30

不改模型权重,只靠自然选择式的方法进化AI智能体的'外壳'(提示词与工具组合),就带来了实质性能提升

DarwinX让LLM的模型权重完全冻结不变,转而把智能体的'外壳(harness)'——提示词、工具、技能文档和控制流程——当作一个变体种群,用类似自然选择的方式筛选。只有在不破坏已解决任务的前提下解决新任务的变体才会被保留,并且会把不同谱系里各自的优势合并起来。在涵盖终端任务、网页自动化和代码修复的四个基准测试中,平均带来约17分的提升。

METAL LAB 解读图

DarwinX选择循环的结构

证据状态已报告实测结果

冻结的模型GPT-5.5、GPT-5.6或Opus 4.8等基座模型权重在整个进化过程中完全不变
生成外壳变体失败分析、教师示范和自我对比信号被转化为对提示词、工具和控制流程的小幅修改
保留并扩展筛选只有在不损害已解决任务的前提下解决新任务的变体才能被提升
存档与重组未被提升的变体也被保留在存档中,不同谱系的互补优势被合并成更强的子变体
四项基准检验检验这种提升在Terminal-Bench 2.1、TerminalWorld、WebArena-Infinity以及迁移到SWE-bench Verified时是否依然成立

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

以往的自我改进智能体沿单一谱系不断修改,容易被早期选择困住,或者修好一类任务却在暗中拖累另一类任务的表现。
DarwinX用一个存档保存一整个外壳变体种群,只有当子变体在不明显损害父代已解决任务的前提下解决了新任务时,才会依据'保留并扩展契约'被提升。
失败分析、教师示范、自身成功/失败轨迹对比这三种学习信号被统一到同一个编辑接口中,用来修改外壳。
评判标准完全来自各基准测试自带的验证器测得的成功率(avg@k),没有标准答案,也没有人工挑选的赢家。
在Terminal-Bench 2.1上,匹配基座模型下从75.5%提升到83.2%(+7.7分),在更强基座上达到84.7%,处于公开验证排行榜最前列;在TerminalWorld的未见过任务集上达到68.3%,超过所有参与对比的现成智能体。

Figure 1: With the base model frozen, evolving the harness alone matches or beats the strongest prior agent on four benchmarks. Left: variants survive on measured fitness (avg@k, no gold solutions) and complementary survivors are merged. Right: bars within a panel share one frozen model; the hatched bar is zero-shot transfer of the Terminal-Bench 2.1 harness. y-ranges are truncated and differ per panel.

Table 1: Positioning among self-improving-agent methods (✓ present, ∼ restricted, ✗ absent). All share the same inner loop and differ in the selection wrapped around it: Search and Selection group the two failure modes we target, path dependence and cross-task interference. Appendix Table 7 gives the mechanism behind each mark.
	Edits	Search	Selection	Signal
Method	tools & control flow	tools &	control flow	population archive	population	archive	cross-lineage merge	cross-lineage	merge	bounded regression	bounded	regression	noise-aware avg@k	noise-aware	avg@k	teacher & self signals	teacher &	self signals
tools &
control flow
population
archive
cross-lineage
merge
bounded
regression
noise-aware
avg@k
teacher &
self signals
Optimizers over one designated artifact
OPRO/PromptBreeder/TextGrad	✗	∼	∼†	✗	✗	✗
ADAS/AFlow/GPTSwarm	∼	✓	✗	✗	∼	✗
SkillOpt	✗	✗	✗	∼	∼	✗
Agents that edit their own scaffold
SICA	✓	∼	✗	✗	✗	✗
DGM	✓	✓	✗	∼	∼	✗
HarnessX	✓	∼	✗	✓	∼	✗
DarwinX (ours)	✓	✓	✓	✓	✓	✓

Figure 2: DarwinX’s selection loop with the model frozen. Left: the preserve-and-extend contract. Middle: the archive of alternative lineages. Right: shared memory carried across generations.

Table 4: WebArena-Infinity per-application audit-clean pass@1 on the official 10-application, 1,260-task real suite; Δ is Monet (DarwinX)’s gain over base Monet. Baseline provenance and raw pre-audit scores are in Appendix D.3.
Application	Kimi	Qwen	Gemini+BU	GPT-5.5+BU	Monet (base)	Monet (DarwinX)	Δ
Elation clinical records	50.0	54.2	81.7	92.5	95.8	96.7	+0.9
Elation prescriptions	23.3	41.7	80.8	90.8	20.0	95.0	+75.0
GitLab plan and track	39.3	37.1	63.6	77.9	63.6	97.9	+34.3
Gmail	70.0	56.7	75.0	85.0	25.0	98.3	+73.3
Gmail accounts and contacts	40.0	33.3	61.7	87.5	21.7	91.7	+70.0
Handshake career exploration	50.0	50.5	50.5	83.5	36.5	84.0	+47.5
Linear account settings	54.2	65.8	73.3	81.7	43.3	94.2	+50.9
PayPal wallet	70.7	71.4	88.6	90.0	49.3	95.7	+46.4
Superhuman general	15.0	25.8	50.0	80.8	31.7	87.5	+55.8
Xero invoicing	52.5	55.8	80.8	93.3	39.2	96.7	+57.5
Overall	43.3	48.3	69.3	86.1	43.5	93.0	+49.5

Figure 3: DarwinX’s per-generation operators. Left: the mutation loop and the three learning signals that drive it. Middle: variants classified by how their solved set changes, where those preserving inherited solves stay eligible for recombination while the rest contribute only distilled lessons. Right: the merge operator and its acceptance criterion.

Table 6: TB2.1 skill-bundle diff: the seven skills the evolved lineage adds over base Monet, all in one verification / artifact-contract family. Skills are co-selected, so this attributes composition, not per-skill effect.
Evolved skills	Role
verifier-contract contract-candidate	Derive the task’s acceptance contract and check the solution against it before finalizing.
graded-artifact-final-check artifact-verification-loop	Verify the graded artifact (output file, format, and values) and iterate a fix-and-recheck loop.
real-tool-artifact tool-grounded-artifact	Ground outputs in real tool execution rather than asserted or simulated results.
security-contract-repair	Repair the solution against security and contract checks.

Figure 7: WebArena-Infinity evolution as optimization. (b) shows the same run as a lineage tree: accepted (blue) and reverted (grey) variants, the primary lineage (gold, base → evolved), and recombination edges (dashed).

Table 7: Mechanism-level comparison with the closest self-improving-agent systems, expanding Table 1. “Promotion rule” is the evidence a candidate must produce before it is kept; “cross-task interference” is how each method prevents an edit that helps one task family from silently regressing another.
Method	Editable surface	Search structure	Promotion rule	Cross-task interference
Optimizers over one designated artifact
OPRO, PromptBreeder, TextGrad	Instruction text; tools and control flow stay fixed.	Iterative keep-best, or a genetic population with prompt crossover.	Scalar score on a fixed development set, or a textual gradient from failures.	Not addressed; the search targets a single task or few-shot pool.
ADAS, AFlow, GPTSwarm	The composition graph over otherwise-fixed components.	Archive of past workflows, or MCTS over graph edits.	Mean accuracy on the target benchmark.	One benchmark per search; no per-task preservation check.
SkillOpt	An external skill document.	Single-lineage keep-best, framed as a gradient-descent analogy.	A held-out validation point estimate gates each edit.	One domain at a time.
Agents that edit their own scaffold
SICA	The agent’s own source code.	A single lineage of self-modification.	Benchmark reward on the current coding suite.	One coding domain; the authors report an early-edit plateau.
DGM	Agent source code.	Open-ended archive, stochastic single-parent mutation, no merge operator.	Score against the parent on a task subset that grows with confidence.	Staged subsets, but no explicit preservation contract.
HarnessX‡	A typed harness: prompts, tools, and control flow.	Staged single-lineage pipeline; variants are kept isolated from each other.	Per-edit gate on the average score plus a seesaw test.	Isolation keeps task families apart, so sub-threshold regressions still accumulate.
DarwinX (ours)	The full harness: a skill layer (prompts, memory, distilled knowledge) and a code layer (tools, control flow, agent loop).	Population archive of typed nodes; parents sampled by cumulative lineage gain; complementary specialists merged into inherited children.	Fitness enabler (g>0, R≤δ) adjudicated by a verifier, then avg@k confirmation and a preservation probe before a node may steer search.	Specialists are retained and recombined rather than isolated, and the preservation probe bounds what any promotion may cost.

(b) Archive lineage tree (node size ∝ screening score).

Table 8: Benchmark-specific base models, evolution and reporting protocols. The base model is frozen throughout every matched comparison, so the harness is the only thing DarwinX changes.
Benchmark	Frozen base	Evolution data	Report data	Selection signal	Report metric
TB2.1	GPT-5.5∘	89 verifier tasks	same 89 tasks	avg@3 screen, avg@5 confirm	avg@5
TerminalWorld	Opus 4.8*	94 train tasks	41 held-out tasks	adaptive avg@k subsets	pass@1
WAI	GPT-5.5	300 synthetic intents	1,260 real tasks	LLM judge, avg@3/avg@5	deterministic pass@1
SWE-V (transfer)	Opus 4.8	none (transfer target)	500 issues	n/a (frozen)	official pass@1

Figure 8: Invalid trajectories before vs. after evolution, by application (left) and mechanism (right). Evolution cuts invalid trajectories from 293 to 17, leaving only raw-state mutations.

Table 10: Construction of the 300-intent WAI evolution split. The benchmark’s own task suites are never read at any stage, so deduplication is intra-pool only.
Stage	Intents	Operation
Seeds	1,080	Two models propose intents from each application’s description document
Raw synthesis	11,279	Dual-model expansion of the seeds under per-application quotas (18 applications)
Filtered	8,013	Quality heuristics, Jaccard near-duplicate removal, TF–IDF uniqueness pruning
Served apps	5,332	Restricted to the 12 applications with a served instance
Evolution split	300	Balanced draw of 25 intents per served application

Figure 9: Per-application success on the real WAI suite, raw pre-audit (left) vs. audit-clean (right). The audit collapses the base’s inflated successes (→43.5%) while leaving Monet (DarwinX) nearly unchanged (→93.0%).

Table 11: Regex patterns used to detect invalid state access or mutation.
#	Regex (paraphrased)	Catches
1	/apistate/	Any read or write of the /api/state scoring endpoint
2	_pushStateToServer \| getSerializableState \| resetToSeedData \| __APP_STATE__ \| window.state=	State-sync internals
3	localStorage/sessionStorage setItem/removeItem/clear	Direct storage writes

Table 12: Examples of valid and invalid agent operations.
Operation	Classification	Reason
Admissible
Click through the UI to add a patient tag	VALID_UI	Normal product operation
Inspect /js/state.js after the app loads or references it	VALID	Client-exposed app knowledge
Read AppState.patients in the browser runtime	VALID	Runtime application observation
POST /patients/123/tags with the current user’s session	VALID_PRODUCT_API	Domain-level app API
AppState.addPatientTag(123, "Flu-Season")	VALID_INTERNAL_APP_COMMAND	App-defined semantic mutator
AppState.currentSection = "patients" to navigate	VALID	Internal navigation; no scored-state fabrication
Inadmissible
cat <apps>/.../js/state.js via the filesystem	INVALID_PRIVILEGED_KNOWLEDGE	Learns from the co-located source tree
Read .../server.py to discover hidden endpoints	INVALID_PRIVILEGED_KNOWLEDGE	Host-only server details
Fetch server source through path traversal or unintended static exposure	INVALID_EXPLOIT_OR_PRIVILEGE_ESCALATION	Browser use does not legitimize an unintended disclosure
patient.tags.push("Flu-Season"); AppState.notify()	INVALID_RAW_STATE_MUTATION	Directly edits the scored representation
Replace local storage with hand-edited serialized state	INVALID_RAW_STATE_MUTATION	Bypasses application semantics
Send a fabricated full state to /api/state	INVALID_EVALUATION_PLANE_ACCESS	Uses benchmark synchronization as a control channel
sqlite3 app.db ’UPDATE patients ...’	INVALID_DIRECT_DATABASE_MANIPULATION	Host access plus a direct database edit
Inject SQL through a request to bypass authorization or alter rows	INVALID_EXPLOIT_OR_PRIVILEGE_ESCALATION	Exploit rather than authorized semantics
Forge an admin token and call an otherwise valid API	INVALID_EXPLOIT_OR_PRIVILEGE_ESCALATION	Exceeds the current user’s authority

Table 13: Contract-oriented browser skills added by evolution.
Skill	When to use	Core procedure
web_task_contract	General browser/web-UI tasks; durable state change or exact filtered/sorted/read answer	Derive an explicit acceptance contract (target, operation, exact final observable state, forbidden substitutions, persistence) → find a valid path, UI first → execute the smallest change → verify both rendered UI and backing state, and reload to confirm persistence → if a target seems missing, prove “not found” from ≥2 independent app surfaces before declaring a no-op.
filtered_list_report_contract	Count / latest / oldest / value questions over lists and tables	Preserve the active collection scope (tab, status, search, project, date range) while applying the requested filter; count across the whole scoped set (not just the rendered page); answer with only the requested value.
browser_spa_state_contract	Durable state changes where visible controls are missing/ambiguous	Derive the exact field-level contract; inspect app-owned stores/reducers/action helpers; mutate through the app’s own action/persistence path; then read back both state and UI.
browser_config_contract	Durable configuration records (filters, rules, reminders, routing)	Prove every field (condition, action, enabled, timing, channel, persistence), not a partial visible match.

Table 14: The evolved browser prompt replaces an absolute UI-only rule with a bounded semantic fallback and persistence verification.
Aspect	Base prompt (before)	Evolved prompt (after)
Interaction policy	“Interact ONLY through the UI…Do NOT write application state directly or touch /api/state.”	“Prefer real UI controls first…If a bounded audit proves no visible UI path can satisfy a durable state-changing task, you may inspect app-owned stores, reducers, loaded modules, public helper methods, and readback paths, then use the app’s own exposed action/update helper for the smallest targeted mutation. Do NOT touch /api/state, write local/session storage, use seed/reset helpers, or call state-sync internals.”
Finishing (verification)	“For a state-changing task, make the change in the UI, screenshot to confirm, then stop.”	“For a state-changing task, verify both app-owned state/readback and the rendered UI; reload or navigate away/back to confirm persistence, then stop.”

研究结果

在Terminal-Bench 2.1(89个任务)上,GPT-5.5基座下未进化的Monet为75.5%,DarwinX进化后达到83.2%(+7.7分),在更强的GPT-5.6基座上达到84.7%,与公开验证排行榜领先者相当或更优。
在TerminalWorld中,94个训练任务之外、彼此不重叠的41个留出任务上,基于Opus 4.8的Monet(DarwinX)解决了28/41(68.3%),超过所有参与评测的现成智能体,相比未进化基座的25/41提升了+7.3分。
在WebArena-Infinity上,仅用300个合成意图进行进化,却使1,260个未见过的真实任务的pass@1在审计后的口径下从43.5%提升到93.0%。
在Terminal-Bench 2.1上进化得到的外壳被原样迁移到SWE-bench Verified(500个议题)上,也取得了实测表现。
提交审计未发现外壳层面的验证器作弊行为,进化前后无效轨迹数从293条降到17条。

可应用场景

为代码或终端类智能体设计通过进化提示词、工具集和控制流程来提升能力的流水线,而非重新训练基座模型。
希望把评估阶段消耗的计算资源转化为持久能力提升、且不想改动模型权重的团队可以参考此思路。
在任何拥有自带自动验证器的任务领域中构建无需标准答案的智能体改进闭环。

局限与待验证事项

该方法需要基准测试自带自动验证器,作者也承认真实部署环境中这类验证器往往并不存在。
avg@k评分方式需要对每个候选变体多次重复运行,适合作为周期性离线任务,但难以支撑按请求实时进化。
模型与外壳协同进化、把保留约束推广到合规性要求等方向,论文只提出设想,尚未做实验验证。
外壳在更换基座模型代际后能保留多少、以及从'热存档'重新选择需要多少代,目前尚未测量。
Terminal-Bench 2.1的技能组合归因反映的是共同被选中的技能组合的整体效果,而非单个技能的独立因果贡献。

为什么重要

这项工作表明,不重新训练模型权重,仅靠自然选择式地筛选更好的提示词和工具组合,也能获得实质性的能力提升,这对希望低成本改进智能体的人有直接参考价值。它还提示外壳可以作为跨模型代际延续的资产,从而降低模型升级带来的重复改造成本。

本文术语

外壳(harness) · 围绕并指挥LLM的提示词、工具、技能文档、记忆和控制流程
avg@k · 对同一任务重复运行k次并取平均成功率的评分方式
保留并扩展契约 · 只有在不明显损失已解决任务的情况下解决新任务的变体才会被提升的规则
重组(recombination) · 把两条不同外壳谱系的修改合并成一个继承双方优点的子变体
pass@1 · 衡量单次尝试是否成功完成任务的指标

论文原文摘要（英文）

An LLM agent's capability depends not only on model weights but on its harness: prompts, tools, skills, and control flow. Self-improvement loops already edit harnesses, yet single-lineage search is path-dependent and local wins often regress other tasks. We introduce DarwinX, which treats self-evolution as selection over a population of harnesses with the model frozen: a preserve-and-extend contract admits only variants that extend coverage without regressing, an archive keeps alternative lineages for recombination, and failure-, teacher-, and self-derived evidence share one edit interface. Fitness comes from each benchmark's own verifier: no gold solutions, no hand-picked winners. Across four benchmarks that progressively separate the evolution signal from the test, one loop adds about 17 points on average: Terminal-Bench 2.1 rises +7.7 to 83.2% on a matched base and to the verified frontier at 84.7% on a stronger one; TerminalWorld's held-out split reaches 68.3%, ahead of every off-the-shelf agent; WebArena-Infinity real-task pass@1 rises from 43.5% to 93.0% audit-clean; and a Terminal-Bench 2.1 harness transfers unchanged to SWE-bench Verified. What evolves is general agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model. A frozen model need not be a fixed agent: harness selection turns evaluation compute into durable capability.

作者 · Yifan Zhang

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Yifan Zhang et al., arXiv:2608.07545, CC BY 4.0