AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

arXiv:2608.255932026-08-27

An AI system that writes a custom 'operating scaffold' for other AI agents on the spot, for every new task

How well an AI agent performs depends less on the language model itself and more on the 'harness' wrapped around it - the rules for memory, planning, tool use, and action. JIT-Agent is a 27B-parameter model trained to generate, repair, and evolve this harness on the fly for any off-the-shelf agentic LLM, instead of relying on hand-built, task-specific scaffolds. With JIT-Agent's generated harness, DeepSeek-V4-Flash surpasses GPT-5.6 on several benchmarks, and an already strong model, GLM-5.2, gains up to 20.2 points.

METAL LAB explanatory visual

JIT-Agent's three-stage training and inference flow

Evidence statusMeasured results reported

  1. Four-module harness protocolHarnesses are standardized into Memory, Planning, Action, and Capability orchestration modules; HarnessFactory reimplements 13 representative existing scaffolds as a seed bank.
  2. Stage I: customization learningThe model imitates a stronger teacher's task-adapted harness examples and learns to prefer harnesses that are both more effective and more efficient.
  3. Stage II: repair learningWhen a generated harness fails to run, the model learns from compiler errors, interface mismatches, and runtime exceptions to fix it within at most two repair rounds.
  4. Stage III: Evo-GDPO online evolutionNew candidate harnesses are compared against the current best archive entry on reward, latency, and cost; only frontier-improving designs are kept, via reinforcement learning.
  5. DeploymentGiven a new task, the trained JIT-Agent generates a custom harness on the spot to wrap any off-the-shelf LLM, and keeps updating it from execution feedback.
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. Problem: agent performance is jointly determined by the model and its 'harness' (memory management, planning strategy, action protocol, tool orchestration), but harnesses have so far been designed manually and separately for each task, which does not scale.
  2. Proposal: JIT-Agent formalizes a harness as four modules - Memory, Planning, Action, and Capability orchestration - under one fixed protocol, and is trained to generate a task-specific harness as executable code just-in-time, whenever a new task arrives.
  3. Training method: a three-stage pipeline. Stage I imitates a stronger teacher model's task-adapted harness examples; Stage II learns to repair harnesses that fail to run by reading diagnostic error reports; Stage III (Evo-GDPO) trains the model online to propose harnesses that beat prior designs in reward, latency, and cost.
  4. Measured results: DeepSeek-V4-Flash with a JIT-Agent-generated harness beats GPT-5.6 by +9.1 on DeepSearchQA, +8.7 on PinchBench, and +4.3 on OdysseyBench; GLM-5.2 gains +12.0 on xBench-DS and +6.9 on AgentIF, with gains up to +20.2 points overall. JIT-Agent-generated harnesses are also reported to be performance-competitive with mature runtimes like OpenCode and Claude Code, and consistently improve differently-sized models within the DeepSeek V4, Mimo-V2.5, and Qwen3.6 families.
  5. Infrastructure: the authors built HarnessFactory, a codebase reimplementing 13 representative existing agent scaffolds (e.g., ReAct, ROMA) under the shared protocol, used as source material and reference bank for JIT-Agent.
Figure 1: Leaderboard across four representative agent benchmarks. JIT-Agent-generated harnesses consistently improve the underlying backbone agents across deep research, daily work, planning, and workspace tasks, demonstrating that just-in-time harness synthesis provides gains beyond model scaling alone.
Figure 1: Leaderboard across four representative agent benchmarks. JIT-Agent-generated harnesses consistently improve the underlying backbone agents across deep research, daily work, planning, and workspace tasks, demonstrating that just-in-time harness synthesis provides gains beyond model scaling alone.
Figure 2: Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by instantiating (rather than simply combining) four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations, as illustrated by the specialized harnesses for deep research, product generation, and autonomous research.
Figure 2: Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by instantiating (rather than simply combining) four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations, as illustrated by the specialized harnesses for deep research, product generation, and autonomous research.
Table 1: Harness optimization paradigms. Construction distinguishes harnesses found by ahead-of-time search, ahead-of-time harnesses subsequently edited with test-time feedback, and harnesses generated just in time for each task. Instance synthesis, Harness model, Learned repair, and Online evolution indicate whether a method directly synthesizes an instance-specific harness, trains the generator, learns repair from failed execution trajectories, and continues improving after deployment, respectively.
MethodConstructionInstance synthesisHarness modelLearned repairOnline evolution
AutoHarness (Lou et al., 2026)AOT (search)\xmark\xmark\xmark\xmark
Meta-Harness (Lee et al., 2026c)AOT (search)\xmark\xmark\xmark\xmark
AHE (Lin et al., 2026)AOT (search)\xmark\xmark\xmark\xmark
Adaptive AH (Liu et al., 2026c)AOT (test-time editing)\xmark\xmark\xmark\cmark
TTHE (Nie et al., 2026b)AOT (test-time editing)\xmark\xmark\xmark\cmark
RHI (Lee et al., 2026a)AOT (test-time editing)\xmark\xmark\xmark\cmark
Harness-R1 (Shao et al., 2026)AOT (test-time editing)\xmark\cmark\cmark\cmark
JIT-Agent (ours)JIT\cmark\cmark\cmark\cmark
Figure 3: Training pipeline of JIT-Agent. Stage I learns task-conditioned harness customization. Stage II converts failed harnesses and execution diagnostics into bounded repair trajectories. Stage III performs online evolution by comparing candidate harnesses against the incumbent bank, optimizing decoupled reward, latency, and cost advantages, and retaining frontier-improving designs.
Figure 3: Training pipeline of JIT-Agent. Stage I learns task-conditioned harness customization. Stage II converts failed harnesses and execution diagnostics into bounded repair trajectories. Stage III performs online evolution by comparing candidate harnesses against the incumbent bank, optimizing decoupled reward, latency, and cost advantages, and retaining frontier-improving designs.
Figure 4: Cost–performance trade-offs on DeepSearchQA and AgentIF. Marker color identifies the harness, while filled and open circles distinguish DeepSeek-V4-Flash and Qwen3.6-Flash. The horizontal axis is API cost per case in USD and the vertical axis is task performance. The dark-green step line traces the global Pareto frontier, and the pale yellow-green and blue-green regions contain pairings dominated by at least one Pareto-optimal point.
Figure 4: Cost–performance trade-offs on DeepSearchQA and AgentIF. Marker color identifies the harness, while filled and open circles distinguish DeepSeek-V4-Flash and Qwen3.6-Flash. The horizontal axis is API cost per case in USD and the vertical axis is task performance. The dark-green step line traces the global Pareto frontier, and the pale yellow-green and blue-green regions contain pairings dominated by at least one Pareto-optimal point.
Table 2: Seed bank ℬ0: 13 hand-written harnesses instantiating the four-module protocol 𝚷. Each row is a complete harness; columns follow the conceptual order of memory, planning, action, and capability orchestration.
♣ Memory♠ Planning♥ Action♠ Capability Orchestration
Harness𝐌∈𝔐𝐏∈𝔓𝐀∈𝔄𝐅∈𝔉
ReAct [83]FullHistoryNo explicit plannerReActFull registry
Plan-and-Execute [16]FullHistoryLinear roadmapReActFull registry
ReSum [77]ReSum memoryNo explicit plannerReActFull registry
Flash-Searcher [60]FullHistoryDAG planningReActFull registry
GAM [79]GAM retrievalDAGReActFull registry
MemoBrain [59]Reasoning graphNo explicit plannerMarker-guided executionFull registry
AggAgent [32]Isolated rollout historiesNo explicit plannerMulti-rollout aggregationFull registry
OAgent [100]Coordinator historyNo explicit plannerEnsemble votingFull registry
AgentFold [84]AgentFold memoryDAGReActFoldFull registry
HiAgent [22]Hierarchical memoryNo explicit plannerReActFull registry
DeepAgent [34]Three-tier memoryNo explicit plannerMarker-guided executionTool search
ROMA [1]Context isolationAtomizer + DAGRecursive executionFull registry
AOrchestra [63]Context isolationAtomizer + DAGRecursive executionAgent Delegation
Figure 5: JIT-generated harnesses consistently improve paired backbones over ReAct. Rows group three model families and two variants within each family; columns correspond to DeepSearchQA, AgentIF-Oneday, DeepPlanning-Shopping, and OfficeBench. Each panel compares the same backbone under a fixed ReAct harness (dashed) and a JIT-generated harness (solid), with callouts reporting the absolute score gain. DeepSearchQA uses a 100-example subset; the other three benchmarks use 50-example subsets.
Figure 5: JIT-generated harnesses consistently improve paired backbones over ReAct. Rows group three model families and two variants within each family; columns correspond to DeepSearchQA, AgentIF-Oneday, DeepPlanning-Shopping, and OfficeBench. Each panel compares the same backbone under a fixed ReAct harness (dashed) and a JIT-generated harness (solid), with callouts reporting the absolute score gain. DeepSearchQA uses a 100-example subset; the other three benchmarks use 50-example subsets.
Figure 6: Streaming test-time harness evolution across task streams. Cumulative accuracy (top), per-task API cost (middle), and per-task tool calls (bottom) on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench. Dashed curves represent Static JIT, where task-specific harness generations are independent, while solid curves represent Streaming JIT, which continuously incorporates execution feedback as new tasks arrive. Shaded regions show local variation around the streaming trajectories. Streaming JIT finishes with higher cumulative accuracy on all three benchmarks, while API-cost and tool-use trajectories remain task-dependent and of broadly similar scale.
Figure 6: Streaming test-time harness evolution across task streams. Cumulative accuracy (top), per-task API cost (middle), and per-task tool calls (bottom) on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench. Dashed curves represent Static JIT, where task-specific harness generations are independent, while solid curves represent Streaming JIT, which continuously incorporates execution feedback as new tasks arrive. Shaded regions show local variation around the streaming trajectories. Streaming JIT finishes with higher cumulative accuracy on all three benchmarks, while API-cost and tool-use trajectories remain task-dependent and of broadly similar scale.

Findings

  • DeepSeek-V4-Flash with a JIT-Agent harness outperformed GPT-5.6 by +9.1 on DeepSearchQA, +8.7 on PinchBench, and +4.3 on OdysseyBench.
  • On the already strong GLM-5.2, JIT-Agent produced up to +20.2 absolute points of gain, including +12.0 on xBench-DS and +6.9 on AgentIF.
  • JIT-Agent-generated harnesses were reported as performance-competitive with mature agent runtimes such as OpenCode and Claude Code.
  • Across differently-scaled model families (DeepSeek V4, Mimo-V2.5, Qwen3.6), JIT-generated harnesses consistently improved over a fixed ReAct baseline harness, and moved backbone-harness pairs toward better cost-performance frontiers on benchmarks like DeepSearchQA and AgentIF.
Figure 7: Palimpsest: graph-planned artifact execution. GraphPlanPlanning converts the contact-processing request into a DAG; GraphPlanAction executes it with bounded width and depth, while GraphPlanMemory stores reusable artifacts and reasoning state.
Figure 7: Palimpsest: graph-planned artifact execution. GraphPlanPlanning converts the contact-processing request into a DAG; GraphPlanAction executes it with bounded width and depth, while GraphPlanMemory stores reusable artifacts and reasoning state.
Figure 8: Trapdoor: bounded research behind a delegated tool call. A synthesized delegate capability is intercepted by OrchestratorLoop, which runs a bounded subagent and writes extracted facts into FactGraphMemory.
Figure 8: Trapdoor: bounded research behind a delegated tool call. A synthesized delegate capability is intercepted by OrchestratorLoop, which runs a bounded subagent and writes extracted facts into FactGraphMemory.

Where it can be used

  • Attaching an automatically generated, task-tailored harness to an existing off-the-shelf LLM (open or commercial) instead of manually building a separate scaffold for each new type of task, such as deep research, shopping planning, travel planning, or office document work.
  • Automating harness construction in pipelines that must handle many different task types with one backbone model, without hand-engineering a new scaffold each time.
  • Using automatic diagnostic-driven repair of failed harness generations as a reference pattern for building more reliable agentic automation systems.
Figure A.1: Origami: hierarchical work with folded context. ROMAPlanning creates isolated subtasks; HierarchicalMemory retains their trajectories and artifacts while fold_thought replaces only the active working context before aggregation.
Figure A.1: Origami: hierarchical work with folded context. ROMAPlanning creates isolated subtasks; HierarchicalMemory retains their trajectories and artifacts while fold_thought replaces only the active working context before aggregation.
Figure A.2: Turnstile: evidence first, itinerary second. TravelPlanning emits a travel specification and checklist, DataStoreMemory tracks the required evidence buckets, and DynamicToolPolicy exposes final_answer only after is_complete() succeeds.
Figure A.2: Turnstile: evidence first, itinerary second. TravelPlanning emits a travel specification and checklist, DataStoreMemory tracks the required evidence buckets, and DynamicToolPolicy exposes final_answer only after is_complete() succeeds.

Limits and open work

  • Reported numbers are measured on specific benchmarks (DeepSearchQA, OdysseyBench, PinchBench, xBench-DS, AgentIF) with specific model backbones (e.g., DeepSeek-V4-Flash, GLM-5.2), and generalization to other tasks or models is not established.
  • Some benchmark comparisons use subsets of only 50 or 100 examples, so absolute gains may differ on the full benchmark distributions.
  • Comparisons to OpenCode and Claude Code are described only as 'performance-competitive,' not as consistently exceeding them.
  • The authors themselves note this work explores a deliberately radical form where the entire harness can be redesigned just-in-time, and suggest future production systems may instead keep a stable core with selective component replacement - a direction the paper does not test.
Figure A.3: Gearbox: one phase shift rewrites the agent. PhaseAction is the sole writer of a shared phase register; PhaseToolPolicy and PhaseAwareMemory read that state to switch both exposed capabilities and typed memory schemas.
Figure A.3: Gearbox: one phase shift rewrites the agent. PhaseAction is the sole writer of a shared phase register; PhaseToolPolicy and PhaseAwareMemory read that state to switch both exposed capabilities and typed memory schemas.
Figure A.4: Pegboard: research as an evidence matrix. Every observation is extracted into a candidate × clue cell with a document identifier; matrix coverage drives both DynamicResearchPlanning and the transition to source-grounded verification.
Figure A.4: Pegboard: research as an evidence matrix. Every observation is extracted into a candidate × clue cell with a document identifier; matrix coverage drives both DynamicResearchPlanning and the transition to source-grounded verification.

Why it matters

Rather than only scaling up the underlying model, this work shows that the scaffold surrounding a model can be treated as a separately trainable and improvable source of capability, offering gains that don't require touching model weights at all. For practitioners, it points to a way of boosting task performance and cost-efficiency by auto-generating a tailored execution structure around an existing off-the-shelf LLM instead of manually engineering one for every new task.

Terms in this paper

  • Agent harness · The operational scaffold around a language model that decides how memory is kept, how plans are formed, which tools are exposed, and how actions are executed
  • Just-in-Time (JIT) harness · Generating a task-specific harness at the moment a task arrives, instead of pre-building one general-purpose harness ahead of time
  • Evo-GDPO · The paper's online reinforcement-learning method that trains the model to propose harnesses that beat the current best archive entry on reward, latency, and cost
  • ReAct · A standard baseline agent loop that alternates between reasoning and acting

Original abstract (English)

Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliable execution, and self-evolve by distilling performance signals from an expanding archive of prior harness configurations. Equipped with JIT-Agent as a harness helper, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3), while the already strong GLM-5.2 gains up to +20.2 points. Across controlled evaluations, JIT-Agent-generated harnesses are performance-competitive with mature agent runtimes such as OpenCode and Claude Code and consistently improve multi-scale model families of DeepSeek V4, Mimo-V2.5, and Qwen3.6. To our knowledge, JIT-Agent is the first model purpose-built for just-in-time harness generation, establishing harness intelligence as a trainable, transferable, and compounding dimension of agent capability orthogonal to model scaling.

Authors · Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Guibin Zhang et al., arXiv:2608.25593, CC BY 4.0