工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

arXiv:2608.255932026-08-27

AI智能体的实力不只取决于模型本身,还取决于包裹模型的'执行框架',这项研究训练了一个能为每个新任务即时生成该框架的AI

AI智能体的表现不仅由语言模型本身决定,更取决于包裹它的'harness'(执行框架)——即记忆管理、规划策略、行动规则和工具调度方式。JIT-Agent是一个270亿参数的模型,能针对任意现成的智能体大模型,在任务到来的那一刻即时生成、修复并持续进化专属执行框架,而不再依赖人工为每类任务单独设计。搭配JIT-Agent生成的执行框架后,DeepSeek-V4-Flash在多个基准测试上超越了GPT-5.6,原本已经很强的GLM-5.2最多也获得了20.2分的提升。

METAL LAB 解读图

JIT-Agent的三阶段训练与运行流程

证据状态已报告实测结果

  1. 四模块执行框架协议将执行框架统一为记忆(Memory)、规划(Planning)、行动(Action)、能力调度(Capability)四个固定模块,并用HarnessFactory重新实现13种代表性现有框架作为初始素材库。
  2. 第一阶段:定制生成学习模仿更强教师模型生成的任务适配框架示例,并学习优先选择在效果和效率上同时更优的框架。
  3. 第二阶段:修复学习当生成的框架运行失败时,模型根据编译错误、接口不匹配、运行异常等诊断信息,在最多两轮修复内学会将其修复为可执行状态。
  4. 第三阶段:Evo-GDPO在线进化将新候选框架与档案库中当前最佳框架在奖励、延迟、成本上比较,只保留能突破现有前沿的设计,通过强化学习持续优化。
  5. 推理部署训练完成的JIT-Agent在新任务到来时即时生成专属执行框架,包裹任意现成大模型执行任务,并根据执行反馈持续更新。
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 问题:智能体的能力由'模型+执行框架'共同决定,但过去执行框架都是人工针对具体任务手工设计的,难以规模化扩展。
  2. 方法:JIT-Agent将执行框架统一形式化为四个固定模块——记忆(Memory)、规划(Planning)、行动(Action)、能力调度(Capability),并训练模型在任务到来时即时生成对应的可执行代码框架。
  3. 训练流程分三阶段:第一阶段模仿更强教师模型生成的任务适配框架示例;第二阶段学习根据执行失败时的诊断报告(编译错误、接口不匹配等)修复框架;第三阶段(Evo-GDPO)通过在线强化学习,让模型不断提出在奖励、延迟和成本上超越已有框架的新设计。
  4. 实测结果:搭配JIT-Agent生成框架的DeepSeek-V4-Flash在DeepSearchQA上比GPT-5.6高9.1分,在PinchBench上高8.7分,在OdysseyBench上高4.3分;本已很强的GLM-5.2在xBench-DS上提升12.0分,在AgentIF上提升6.9分,整体最高提升达20.2分。此外,JIT-Agent生成的框架据报告与OpenCode、Claude Code等成熟智能体运行环境表现相当,并在DeepSeek V4、Mimo-V2.5、Qwen3.6等不同规模的模型家族上均带来一致提升。
  5. 配套设施:研究者构建了HarnessFactory代码库,在统一协议下重新实现了13种具有代表性的现有智能体框架(如ReAct、ROMA),作为JIT-Agent生成时可参考的素材库。
Figure 1: Leaderboard across four representative agent benchmarks. JIT-Agent-generated harnesses consistently improve the underlying backbone agents across deep research, daily work, planning, and workspace tasks, demonstrating that just-in-time harness synthesis provides gains beyond model scaling alone.
Figure 1: Leaderboard across four representative agent benchmarks. JIT-Agent-generated harnesses consistently improve the underlying backbone agents across deep research, daily work, planning, and workspace tasks, demonstrating that just-in-time harness synthesis provides gains beyond model scaling alone.
Figure 2: Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by instantiating (rather than simply combining) four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations, as illustrated by the specialized harnesses for deep research, product generation, and autonomous research.
Figure 2: Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by instantiating (rather than simply combining) four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations, as illustrated by the specialized harnesses for deep research, product generation, and autonomous research.
Table 1: Harness optimization paradigms. Construction distinguishes harnesses found by ahead-of-time search, ahead-of-time harnesses subsequently edited with test-time feedback, and harnesses generated just in time for each task. Instance synthesis, Harness model, Learned repair, and Online evolution indicate whether a method directly synthesizes an instance-specific harness, trains the generator, learns repair from failed execution trajectories, and continues improving after deployment, respectively.
MethodConstructionInstance synthesisHarness modelLearned repairOnline evolution
AutoHarness (Lou et al., 2026)AOT (search)\xmark\xmark\xmark\xmark
Meta-Harness (Lee et al., 2026c)AOT (search)\xmark\xmark\xmark\xmark
AHE (Lin et al., 2026)AOT (search)\xmark\xmark\xmark\xmark
Adaptive AH (Liu et al., 2026c)AOT (test-time editing)\xmark\xmark\xmark\cmark
TTHE (Nie et al., 2026b)AOT (test-time editing)\xmark\xmark\xmark\cmark
RHI (Lee et al., 2026a)AOT (test-time editing)\xmark\xmark\xmark\cmark
Harness-R1 (Shao et al., 2026)AOT (test-time editing)\xmark\cmark\cmark\cmark
JIT-Agent (ours)JIT\cmark\cmark\cmark\cmark
Figure 3: Training pipeline of JIT-Agent. Stage I learns task-conditioned harness customization. Stage II converts failed harnesses and execution diagnostics into bounded repair trajectories. Stage III performs online evolution by comparing candidate harnesses against the incumbent bank, optimizing decoupled reward, latency, and cost advantages, and retaining frontier-improving designs.
Figure 3: Training pipeline of JIT-Agent. Stage I learns task-conditioned harness customization. Stage II converts failed harnesses and execution diagnostics into bounded repair trajectories. Stage III performs online evolution by comparing candidate harnesses against the incumbent bank, optimizing decoupled reward, latency, and cost advantages, and retaining frontier-improving designs.
Figure 4: Cost–performance trade-offs on DeepSearchQA and AgentIF. Marker color identifies the harness, while filled and open circles distinguish DeepSeek-V4-Flash and Qwen3.6-Flash. The horizontal axis is API cost per case in USD and the vertical axis is task performance. The dark-green step line traces the global Pareto frontier, and the pale yellow-green and blue-green regions contain pairings dominated by at least one Pareto-optimal point.
Figure 4: Cost–performance trade-offs on DeepSearchQA and AgentIF. Marker color identifies the harness, while filled and open circles distinguish DeepSeek-V4-Flash and Qwen3.6-Flash. The horizontal axis is API cost per case in USD and the vertical axis is task performance. The dark-green step line traces the global Pareto frontier, and the pale yellow-green and blue-green regions contain pairings dominated by at least one Pareto-optimal point.
Table 2: Seed bank ℬ0: 13 hand-written harnesses instantiating the four-module protocol 𝚷. Each row is a complete harness; columns follow the conceptual order of memory, planning, action, and capability orchestration.
♣ Memory♠ Planning♥ Action♠ Capability Orchestration
Harness𝐌∈𝔐𝐏∈𝔓𝐀∈𝔄𝐅∈𝔉
ReAct [83]FullHistoryNo explicit plannerReActFull registry
Plan-and-Execute [16]FullHistoryLinear roadmapReActFull registry
ReSum [77]ReSum memoryNo explicit plannerReActFull registry
Flash-Searcher [60]FullHistoryDAG planningReActFull registry
GAM [79]GAM retrievalDAGReActFull registry
MemoBrain [59]Reasoning graphNo explicit plannerMarker-guided executionFull registry
AggAgent [32]Isolated rollout historiesNo explicit plannerMulti-rollout aggregationFull registry
OAgent [100]Coordinator historyNo explicit plannerEnsemble votingFull registry
AgentFold [84]AgentFold memoryDAGReActFoldFull registry
HiAgent [22]Hierarchical memoryNo explicit plannerReActFull registry
DeepAgent [34]Three-tier memoryNo explicit plannerMarker-guided executionTool search
ROMA [1]Context isolationAtomizer + DAGRecursive executionFull registry
AOrchestra [63]Context isolationAtomizer + DAGRecursive executionAgent Delegation
Figure 5: JIT-generated harnesses consistently improve paired backbones over ReAct. Rows group three model families and two variants within each family; columns correspond to DeepSearchQA, AgentIF-Oneday, DeepPlanning-Shopping, and OfficeBench. Each panel compares the same backbone under a fixed ReAct harness (dashed) and a JIT-generated harness (solid), with callouts reporting the absolute score gain. DeepSearchQA uses a 100-example subset; the other three benchmarks use 50-example subsets.
Figure 5: JIT-generated harnesses consistently improve paired backbones over ReAct. Rows group three model families and two variants within each family; columns correspond to DeepSearchQA, AgentIF-Oneday, DeepPlanning-Shopping, and OfficeBench. Each panel compares the same backbone under a fixed ReAct harness (dashed) and a JIT-generated harness (solid), with callouts reporting the absolute score gain. DeepSearchQA uses a 100-example subset; the other three benchmarks use 50-example subsets.
Figure 6: Streaming test-time harness evolution across task streams. Cumulative accuracy (top), per-task API cost (middle), and per-task tool calls (bottom) on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench. Dashed curves represent Static JIT, where task-specific harness generations are independent, while solid curves represent Streaming JIT, which continuously incorporates execution feedback as new tasks arrive. Shaded regions show local variation around the streaming trajectories. Streaming JIT finishes with higher cumulative accuracy on all three benchmarks, while API-cost and tool-use trajectories remain task-dependent and of broadly similar scale.
Figure 6: Streaming test-time harness evolution across task streams. Cumulative accuracy (top), per-task API cost (middle), and per-task tool calls (bottom) on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench. Dashed curves represent Static JIT, where task-specific harness generations are independent, while solid curves represent Streaming JIT, which continuously incorporates execution feedback as new tasks arrive. Shaded regions show local variation around the streaming trajectories. Streaming JIT finishes with higher cumulative accuracy on all three benchmarks, while API-cost and tool-use trajectories remain task-dependent and of broadly similar scale.

研究结果

  • 搭配JIT-Agent框架的DeepSeek-V4-Flash相比GPT-5.6,在DeepSearchQA上高出9.1分,在PinchBench上高出8.7分,在OdysseyBench上高出4.3分。
  • 在本已表现强劲的GLM-5.2上应用JIT-Agent后,xBench-DS提升12.0分,AgentIF提升6.9分,整体最大绝对提升达20.2分。
  • 据报告,JIT-Agent生成的执行框架在性能上与OpenCode、Claude Code等成熟智能体运行环境相当。
  • 在DeepSeek V4、Mimo-V2.5、Qwen3.6等不同规模的模型家族上,JIT生成的框架相较固定的ReAct基线框架均带来一致提升,并在DeepSearchQA、AgentIF等测试上改善了成本与性能的权衡边界。
Figure 7: Palimpsest: graph-planned artifact execution. GraphPlanPlanning converts the contact-processing request into a DAG; GraphPlanAction executes it with bounded width and depth, while GraphPlanMemory stores reusable artifacts and reasoning state.
Figure 7: Palimpsest: graph-planned artifact execution. GraphPlanPlanning converts the contact-processing request into a DAG; GraphPlanAction executes it with bounded width and depth, while GraphPlanMemory stores reusable artifacts and reasoning state.
Figure 8: Trapdoor: bounded research behind a delegated tool call. A synthesized delegate capability is intercepted by OrchestratorLoop, which runs a bounded subagent and writes extracted facts into FactGraphMemory.
Figure 8: Trapdoor: bounded research behind a delegated tool call. A synthesized delegate capability is intercepted by OrchestratorLoop, which runs a bounded subagent and writes extracted facts into FactGraphMemory.

可应用场景

  • 在不更换现有开源或商用大模型的前提下,针对深度调研、购物规划、旅行规划、办公文档处理等不同类型任务,自动生成定制执行框架来替代人工搭建。
  • 在需要用同一个基础模型处理多种性质迥异任务的自动化流水线中,用自动生成执行框架的方式替代逐一手工设计。
  • 作为参考案例,借鉴其'依据执行失败诊断信息自动修复执行框架'的设计思路,用于提升自动化系统的可靠性。
Figure A.1: Origami: hierarchical work with folded context. ROMAPlanning creates isolated subtasks; HierarchicalMemory retains their trajectories and artifacts while fold_thought replaces only the active working context before aggregation.
Figure A.1: Origami: hierarchical work with folded context. ROMAPlanning creates isolated subtasks; HierarchicalMemory retains their trajectories and artifacts while fold_thought replaces only the active working context before aggregation.
Figure A.2: Turnstile: evidence first, itinerary second. TravelPlanning emits a travel specification and checklist, DataStoreMemory tracks the required evidence buckets, and DynamicToolPolicy exposes final_answer only after is_complete() succeeds.
Figure A.2: Turnstile: evidence first, itinerary second. TravelPlanning emits a travel specification and checklist, DataStoreMemory tracks the required evidence buckets, and DynamicToolPolicy exposes final_answer only after is_complete() succeeds.

局限与待验证事项

  • 论文报告的数值是在DeepSearchQA、OdysseyBench、PinchBench、xBench-DS、AgentIF等特定基准测试以及DeepSeek-V4-Flash、GLM-5.2等特定模型组合上测得的,不能直接断定可推广到其他任务或模型。
  • 部分基准比较仅使用了50或100个样本的子集,与完整数据集上的绝对提升幅度可能有所不同。
  • 与OpenCode、Claude Code的比较仅被描述为'表现相当',并未给出全面超越这些系统的证据。
  • 作者自己也指出,本研究采用了较为激进的形式,即整个执行框架都可以被即时重新设计;而未来的生产系统或许更适合保留稳定核心、只在需要时替换部分组件,这一方向本文尚未验证。
Figure A.3: Gearbox: one phase shift rewrites the agent. PhaseAction is the sole writer of a shared phase register; PhaseToolPolicy and PhaseAwareMemory read that state to switch both exposed capabilities and typed memory schemas.
Figure A.3: Gearbox: one phase shift rewrites the agent. PhaseAction is the sole writer of a shared phase register; PhaseToolPolicy and PhaseAwareMemory read that state to switch both exposed capabilities and typed memory schemas.
Figure A.4: Pegboard: research as an evidence matrix. Every observation is extracted into a candidate × clue cell with a document identifier; matrix coverage drives both DynamicResearchPlanning and the transition to source-grounded verification.
Figure A.4: Pegboard: research as an evidence matrix. Every observation is extracted into a candidate × clue cell with a document identifier; matrix coverage drives both DynamicResearchPlanning and the transition to source-grounded verification.

为什么重要

这项研究表明,提升智能体能力不一定要靠扩大模型规模,包裹模型的执行框架本身也可以作为一个可单独训练、可迁移的能力维度来优化。对实践者而言,这提示了一种思路:无需更换现有的大模型,只需自动生成针对具体任务的定制执行框架,就有可能在性能和成本之间获得更好的平衡。

本文术语

  • 智能体执行框架(harness) · 包裹在语言模型外层、决定如何管理记忆、如何规划、暴露哪些工具、如何执行动作的运行结构
  • 即时生成(Just-in-Time) · 在任务真正到来的那一刻才针对该任务生成专属执行框架,而不是提前造好一个通用框架
  • Evo-GDPO · 论文提出的一种在线强化学习方法,训练模型提出在奖励、延迟、成本上超越当前最佳框架档案的新设计
  • ReAct · 一种交替进行推理与行动的标准基础智能体执行方式

论文原文摘要（英文）

Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliable execution, and self-evolve by distilling performance signals from an expanding archive of prior harness configurations. Equipped with JIT-Agent as a harness helper, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3), while the already strong GLM-5.2 gains up to +20.2 points. Across controlled evaluations, JIT-Agent-generated harnesses are performance-competitive with mature agent runtimes such as OpenCode and Claude Code and consistently improve multi-scale model families of DeepSeek V4, Mimo-V2.5, and Qwen3.6. To our knowledge, JIT-Agent is the first model purpose-built for just-in-time harness generation, establishing harness intelligence as a trainable, transferable, and compounding dimension of agent capability orthogonal to model scaling.

作者 · Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Guibin Zhang et al., arXiv:2608.25593, CC BY 4.0