Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

arXiv:2608.084662026-08-08

Letting an AI keep rewriting the scaffolding around itself, without touching the model, improved its performance

An LLM agent's performance depends not just on the model itself but on the harness, the surrounding code scaffold of prompts, tools, and memory that lets it actually complete tasks. This paper introduces Hierarchical Self-Improvement (HSI), where a single frozen LLM operates at three nested levels to rewrite its own task harness and even the strategy used to rewrite that harness. Tested on the BALROG benchmark with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI improved consistently on moderate-difficulty tasks but showed no gain on a task (NLE) that exceeded the model's own capability.

METAL LAB explanatory visual

Letting an AI keep rewriting the scaffolding around itself, without touching the model, improved its performance

01A single frozen (parameters unchanged) LLM is given three separated roles: executing the task harness, rewriting the harness ('evolver'), and rewriting the evolver's own strategy ('meta-evolver'), with the outermost execution logic kept frozen to prevent uncontrolled self-reference
02Reasoning ('thinking') was turned off during task execution but kept on during self-modification, isolating whether gains came from harness improvements rather than extra inference-time reasoning
03On moderate-difficulty BALROG tasks, raw % Progress gains over the initial harness reached +39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack
04On BabaIsAI, with 20% of tasks held out entirely during evolution, the evolved harness still achieved 0.98 best-test progress on BreakStop and 1.00 on GoTo, showing genuine generalization rather than memorization
05On NLE, a task beyond the frozen model's capability, harness evolution produced no meaningful improvement, revealing a clear ceiling on what harness rewriting alone can fix

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

A single frozen (parameters unchanged) LLM is given three separated roles: executing the task harness, rewriting the harness ('evolver'), and rewriting the evolver's own strategy ('meta-evolver'), with the outermost execution logic kept frozen to prevent uncontrolled self-reference
Reasoning ('thinking') was turned off during task execution but kept on during self-modification, isolating whether gains came from harness improvements rather than extra inference-time reasoning
On moderate-difficulty BALROG tasks, raw % Progress gains over the initial harness reached +39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack
On BabaIsAI, with 20% of tasks held out entirely during evolution, the evolved harness still achieved 0.98 best-test progress on BreakStop and 1.00 on GoTo, showing genuine generalization rather than memorization
On NLE, a task beyond the frozen model's capability, harness evolution produced no meaningful improvement, revealing a clear ceiling on what harness rewriting alone can fix

Figure 1: The HSI framework. A single frozen LLM M operates across three hierarchical scopes with disjoint editable surfaces. The task-harness scope executes the task-specific harness H; the evolver scope rewrites H through seed selection, main evolution, and commit selection; and the meta-evolver scope rewrites the evolver strategy Σ, including seed, evolution, commit, and final version selection policies.

Table 1: BALROG leaderboard comparison under Setup A. The top block lists public-leaderboard numbers (retrieved 2026-08-03) from frontier models under their native configurations, reported as % Progress (15) (mean ± standard deviation across evaluation episodes). The bottom block reports HSI on the same task suite using a single frozen DeepSeek-V4-Flash backbone, isolating the contribution of the continuously evolvable hot-swappable task harness: the init-harness baseline, the meta-evolution-off arm, and the meta-evolution-on arm (bolded) that exports the deployed harness. BabaIsAI is omitted because our sub-suite protocol (§4.2.2) differs from the leaderboard’s mixed-task protocol. Avg reports the unweighted mean across the five environments, with its std the unweighted mean of the per-environment stds.
LLM	BabyAI	Crafter	TextWorld	MiniHack	NLE	Avg
Gemini-3-Pro	96.0±2.8	57.3±4.4	60.2±7.5	40.0±7.7	6.8±3.2	52.1±5.1
Gemini-3.1-Pro-Thinking	98.0±2.0	55.0±6.4	75.7±6.4	27.5±7.1	2.6±0.3	51.8±4.4
Gemini-3.1-Pro	100.0±0.0	46.8±4.2	66.5±7.5	35.0±7.5	3.0±0.5	50.3±3.9
Gemini-3-Flash	86.0±4.9	45.0±6.3	50.2±8.1	30.0±7.2	4.0±0.8	43.0±5.5
Grok-4	76.0±6.0	57.3±3.9	62.9±7.9	17.5±6.0	1.8±0.8	43.1±4.9
Claude-Opus-4.5	80.0±5.7	49.5±3.1	51.4±8.4	27.5±7.1	2.0±0.5	42.1±5.0
Claude-Opus-4.5-Thinking	72.0±6.3	48.6±3.2	59.0±8.0	30.0±7.2	2.4±0.3	42.4±5.0
Gemini-2.5-Pro-Exp-03-25	80.0±5.7	55.0±6.0	49.2±8.2	17.5±6.0	1.7±0.2	40.7±5.2
DeepSeek-R1	74.0±6.2	36.4±3.8	21.8±6.1	25.0±6.8	1.4±0.5	31.7±4.7
GPT-5-minimal-think	80.0±5.7	39.1±4.1	30.6±7.0	20.0±7.3	1.3±0.5	34.2±4.9
Claude-3.5-Sonnet	68.0±6.6	32.7±3.2	42.1±5.4	15.0±5.6	0.6±0.5	31.7±4.3
GPT-4o	77.6±3.7	33.1±2.3	39.3±5.2	10.0±4.7	0.4±0.4	32.1±3.3
DS-V4-Flash (Init harness)	42.0±3.5	11.6±5.0	40.0±6.2	0.8±1.9	0.0	18.9±3.3
DS-V4-Flash w. HSI (meta-off)	77.3±1.2	36.4±1.6	46.0±2.4	5.8±3.8	0.0	33.1±1.8
DS-V4-Flash w. HSI (meta-on)	81.3±4.2	44.6±3.2	65.0±3.0	15.8±2.9	0.2±0.3	41.4±2.7

Figure 2: HSI evolution trajectory on Crafter (Setup A, meta-on). The dev reward climbs from the init-harness baseline 0.166 to an iteration-best 0.578 by iteration 4 (green-starred as the exported best version), with a regression in iteration 5. Each iteration’s annotation card reports four fields on the task-harness and evolver scopes (seed origin, main-evolution edit, result, commit pool) plus a green-tinted meta field summarizing what the meta-evolver rewrote in the evolver strategy Σ that iteration. The dominant lever uncovered by the evolver is making hidden game feedback explicit: reward signal, inventory state, and crafting feasibility, successively exposed in the harness context; the meta-evolver’s contribution is to codify these patterns into Σ so later iterations inherit them.

Table 2: BabaIsAI sub-suite results under Setup B (split evolution with 20% held-out test). “Best Dev” denotes the highest development reward selected during evolution. Test results are reported as mean ± across-task standard deviation of task progress. Init Harness is averaged over three baseline runs.
Sub-suite	Init Harness	Best Dev	Best Test (meta-on)	Best Test (meta-off)
BreakStop	0.0333±0.0334	1.0000	0.9800±0.0632	1.0000±0.0000
GoTo	0.1818±0.0802	1.0000	1.0000±0.0000	0.9636±0.0809
Make	0.0000	0.5556	0.3625±0.3284	0.3375±0.2029

Figure 3: HSI evolution trajectory on BabaIsAI-Make (Setup B, meta-on). The main blue curve is the dev reward across all reward_history evaluations; red rings mark commit-pool versions. Orange triangles (△) flag commits finalized in val mode (no dev evaluation recorded on the commit itself): each △ sits at the commit’s anchor x with the val reward on the y-axis, and a dashed vertical line drops from the dev anchor to the val reward so the dev→val evaluation structure reads directly off the trajectory. The dev peak climbs 0.222→0.333→0.444→0.556 across five iterations as the agent introduces a single-react rewrite with plan tracking (iter 1), a spatial-map builder with WIN-target persistence (iter 2), auto-target computation with BFS pathfinding (iter 3), an auto-push mechanism with directional fallback (iter 4), and LLM-aware cross-room navigation (iter 5); the meta-evolver codifies the “LLM targets, BFS navigates” two-layer pattern and progressively expands the commit pool from one to three versions. The △ markers expose which iterations’ primary commits were validated on held-out val tasks, an evaluation structure unique to Setup B.

Table 3: Per-suite experimental configuration. Shared settings are listed in the prose above; only the entries below vary across suites.
Suite	Setup	Dev	Val	Test ep.	Dev ep.	Test rep.	Meta	Submit-best
TextWorld	A	1.0	0.00	10	3	3	off	50
BabyAI	A	1.0	0.00	10	3	3	on	80
Crafter	A	1.0	0.00	5	3	3	on	50
MiniHack	A	1.0	0.00	5	1	3	on	80
NLE	A	1.0	0.00	5	1	1	on	50
BabaIsAI-BreakStop	B	0.8	0.20	5	1	1	off	80
BabaIsAI-GoTo	B	0.8	0.25	5	1	1	on	80
BabaIsAI-Make	B	0.8	0.25	5	1	1	on	80

Table 4: Representative harness evolution and self-improvement methods compared with HSI. Proposer: who proposes harness edits; Surface: the editable code surface; Domain: primary evaluation domain; Feature: the method’s distinguishing contribution in short form.
Method	Proposer	Surface	Domain	Feature
Meta-Harness	External stronger	Full harness code	Coding, math	Full-trajectory feedback
Self-Harness	Self (target)	Config interface	Coding	Model-specific edits
AHE	External stronger	Decoupled comps	Coding	Observability bottleneck
HarnessX	Multi-agent	Typed processors	5 benchmarks	Operational mirror to RL
DGM	Self	Full codebase	Coding	Archive-based search
HGM	Self	Full codebase	Coding	Clade meta-productivity
GEA	Self (group)	Codebase + exp	Coding	Shared experience pool
SICA	Self	Full codebase	Coding	Framework saturation
HyperAgents	Fused self	Codebase + meta	Coding, robots	Editable meta-mechanism
Live-SWE-Agent	Self (runtime)	Tools on-the-fly	Coding	Zero offline cost
TTHE	Self (test-time)	Harness population	Coding, SQL	Unlabeled trace adaptation
Rethinking Eval.	— (critique)	—	Coding	Test-time-scaling confound
Statistical Limits	— (theory)	—	PAC learning	VC bound ⇔ learnability
HSI (Ours)	Same frozen M	3-layer hierarchy	BALROG	Endogenous hierarchy with frozen outer anchor

Why it matters

Most agent improvements today rely on humans manually tweaking prompts or tools, but this work shows the surrounding scaffold itself can be safely and hierarchically self-modified by the AI without changing model weights. It also empirically pinpoints where such self-improvement stops working, giving practical guidance on when to invest in harness engineering versus upgrading the underlying model.

Terms in this paper

harness · the executable scaffold of prompts, tools, memory, and control logic that wraps around an LLM to run tasks
frozen LLM · a language model whose trained parameters are kept fixed and never updated
evolver / meta-evolver · the role that rewrites the task harness, and the role one level above that rewrites the evolver's own strategy
% Progress · a 0-100 scale metric in the BALROG benchmark measuring how much of a task was completed
held-out generalization · testing on tasks never seen during the improvement process to check if gains transfer to new, unseen cases

Original abstract (English)

Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the harness---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is task-specific and continuously evolvable: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce Hierarchical Self-Improvement (HSI), a framework in which a single frozen LLM M operates across three hierarchical scopes: a task harness H that executes tasks, an evolver that rewrites H, and a meta-evolver that rewrites the evolver's strategy code under a frozen outer anchor. A thinking-on/off design isolates the contribution of harness evolution by disabling reasoning during task execution while enabling it during self-modification. HSI is bounded by two factors: a feedback-fidelity bound, since evolution requires informative reward signals to guide selection, and a backbone capability bound, since harness redesign cannot overcome limitations of the frozen model. On BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI achieves consistent gains over the initial harness on moderate-difficulty tasks (+39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack, all in raw \% Progress), while obtaining strong held-out generalization on BabaIsAI sub-suites (0.98 best-test on BreakStop and 1.00 on GoTo from a 20% unseen split). On tasks beyond the backbone's capability (NLE), harness evolution provides no improvement. These results demonstrate task-specific harness evolution as a viable axis for improving frozen LLM agents under clear empirical limits. Code is available at https://github.com/TailinZhou/hsi.

Authors · Tailin Zhou

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Tailin Zhou et al., arXiv:2608.08466, CC BY 4.0