FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis
arXiv:2608.185802026-08-20
FACET builds internally consistent terminal-task 'exam sets' to train command-line AI agents
Training AI agents to work in a terminal requires large sets of tasks where the instruction, starting environment, reference solution, and automated checker all agree with each other. FACET recombines real-world agent skills into coherent scenarios, builds and fixes a working Docker environment first, then generates the instruction, solution, and verifier in that order so they stay grounded in the same actual files and state. Using just 1,200 successful task trajectories out of 6,078 validated tasks, fine-tuning Qwen3.5 models (4B to 27B) produced consistent gains on Terminal-Bench 2.1.
What they did
Over 71,000 public agent skills were collected and cleaned, then skills sharing a plausible common goal were grouped and reconstructed into a single richer user scenario.
The scenario is described along five angles (goal, context, capability, state, inputs/outputs/tools) to build a solution reference and an instruction reference, which a model then cross-checks for matching initial/final states.
The execution environment (Docker image, files, services) is built first and auto-repaired up to 3 times if it fails; only after it works are the instruction, solution, and verifier generated in that fixed order, all reading from the same realized environment so they reference the same files, paths, and schemas.
A task is only accepted if the environment builds, the verifier rejects the untouched initial state, the reference solution runs successfully, and the verifier then accepts the final state; failures trigger targeted repair (up to 5 rounds) of only the broken component instead of regenerating everything.
Comparing generation orders, doing instruction then solution then verifier (Forward) reached a much higher initial validity rate (46.5%) than generating the verifier before the solution (24.2%); fine-tuning on the resulting 1,200 successful trajectories out of 6,078 validated tasks raised Qwen3.5-4B from 17.60 to 24.72, 9B from 27.34 to 35.58, and 27B from 40.82 to 47.57, nearly matching a 397B model's 49.06 score.
Figure 1: Overview of FACET. Stage 1 collects skills and constructs the scenario–skill repository. Stage 2 understands the selected skills, explores and recovers a joint scenario, expands it into a complete representation, and builds aligned solution and instruction references. Stage 3 constructs and validates the environment before generating the final task artifacts, with bounded repair loops for failed builds and tests.
Table 1: Trajectory- and task-level comparison of terminal-agent datasets. Trajectories are collected using the Terminus-2 scaffold, and task performance is evaluated using DeepSeek-V4-Pro with Terminus-2. Turns and Tests denote the average interaction turns per trajectory and executable checkpoints per task. Detailed protocols are provided in Appendix A.6.
Dataset
Trajectory
Task
#Traj.
Turns
#Tasks
Tests
P@1
P@3
Nemotron-Terminal (29)
5K
6.12
15K
6.18
40.67
48.00
Endless-Terminals (13)
200
4.53
2,492
5.51
83.00
87.00
Terminal-Lego (33)
32K
5.77
15K
16.60
47.00
49.00
TerminalWorld (8)
200
11.94
1,530
3.98
57.00
82.00
Tmax (16)
500
11.14
15K
3.29
80.00
86.00
FACET (ours)
1.2K
11.86
6K
22.77
27.00
35.00
Figure 2: Analysis of execution and synthesis failures. (a) Distribution of failed or errored verifier checks among unsuccessful teacher rollouts; three rollouts without a parsed FAILED/ERROR result are omitted. (b) Distribution of initial validation failure types under the Forward, Reverse, and Joint generation schemes.
Table 2: Results on Terminal-Bench 2.1. Scores under our evaluation setting are averaged over three independent attempts per task.
Model
Size
Agent
Terminal-Bench 2.1
Reported reference models
GPT-5.5 (xhigh) (31)
—
Terminus-2
78.00
Claude Opus 4.7 (max) (31)
—
Terminus-2
66.10
Gemini 3 Pro (high) (31)
—
Gemini CLI
65.80
Intern-S2-Preview-397B (7)
397B
Terminus-2
67.42
MiniMax M3 (19)
428B
Terminus-2
66.00
GLM-5.1 (max) (31)
744B
Claude Code
58.70
Models evaluated under our setting
Qwen3.6-27B
27B
Terminus-2
53.93
Qwen3.5-397B-A17B
397B
Terminus-2
49.06
Kimi-K2.6
1T
Terminus-2
59.93
DeepSeek-V4-Pro-Preview (high)
1.6T
Terminus-2
73.03
Qwen3.5 base models
Qwen3.5-4B
4B
Terminus-2
17.60
Qwen3.5-9B
9B
Terminus-2
27.34
Qwen3.5-27B
27B
Terminus-2
40.82
Fine-tuned models
FACET-Terminal-Qwen3.5-4B
4B
Terminus-2
24.72 (+7.12)
FACET-Terminal-Qwen3.5-9B
9B
Terminus-2
35.58 (+8.24)
FACET-Terminal-Qwen3.5-27B
27B
Terminus-2
47.57 (+6.75)
Figure 3: Distributions of source skills and synthesized tasks. (a) The retained skill corpus spans five top-level families and 34 fine-grained categories. (b) The 6,078 validated tasks are distributed across nine task families, whose individual shares range from 9.59% to 11.99%.
Table 3: Top-level source-skill distribution.
Category
Skills
Share
AI, agents, and tools
15,182
21.28%
Software, systems, and security
15,059
21.11%
Data, analysis, and research
12,409
17.39%
Documents, productivity, and workflows
11,267
15.79%
Multimedia, creation, and publishing
17,424
24.42%
Total
71,341
100.00%
Figure 4: Strict task-level pass rates for the 20 most frequent skill tags among the 6,066 completed rollouts. Error bars indicate Wilson 95% confidence intervals, and the dashed line denotes the overall pass rate. Tags are selected by frequency and ordered by observed pass rate.
Table 4: Fine-grained source-skill categories. “Global” is the share of all 71,341 skills; “within parent” is the share inside the corresponding top-level category.
Parent
Fine-grained category
Count
Global
Within parent
AI, agents, and tools
Agent orchestration and automation
2,782
3.90%
18.32%
Prompt and model calls
3,310
4.64%
21.80%
Skills, plugins, and extensions
1,637
2.29%
10.78%
MCP and external tools
2,579
3.62%
16.99%
Memory, RAG, and knowledge bases
3,439
4.82%
22.65%
Multi-agent collaboration
1,435
2.01%
9.45%
Software, systems, and security
Code generation and development
3,065
4.30%
20.35%
Testing, debugging, and code quality
2,167
3.04%
14.39%
Git, build, and dependency management
1,904
2.67%
12.64%
Deployment, containers, and DevOps
2,069
2.90%
13.74%
System administration and CLI
3,576
5.01%
23.75%
Security, privacy, and compliance
2,278
3.19%
15.13%
Data, analysis, and research
JSON, YAML, and XML
2,752
3.86%
22.18%
CSV, Excel, and spreadsheets
1,544
2.16%
12.44%
Databases and SQL
1,627
2.28%
13.11%
Data cleaning, conversion, and validation
1,595
2.24%
12.85%
Statistical analysis and metrics
1,892
2.65%
15.25%
Visualization and dashboards
1,419
1.99%
11.44%
Search, research, and extraction
1,580
2.21%
12.73%
Documents, productivity, and workflows
Documents and Markdown
1,496
2.10%
13.28%
Reports, summaries, and briefs
1,865
2.61%
16.55%
PDF, Office, and presentations
1,429
2.00%
12.68%
Office and personal productivity
1,592
2.23%
14.13%
Project, task, and schedule management
1,682
2.36%
14.93%
System integration and automation
1,597
2.24%
14.17%
Audit, checklists, and operation records
1,606
2.25%
14.25%
Multimedia, creation, and publishing
Image generation and editing
2,699
3.78%
15.49%
Design, drawing, and visual assets
1,858
2.60%
10.66%
Audio, speech, and music
1,917
2.69%
11.00%
Video, animation, and captions
2,491
3.49%
14.30%
Figure 5: Command-level patterns in successful teacher trajectories. The left panel shows the most frequent shell commands, colored by their dominant contextual class. The right panel reports row-normalized transitions between adjacent assistant-turn states.
Table 5: Task-construction funnel. Percentages in the last column use the immediately preceding comparable stage.
Stage
Count
Stage retention
Scenario–skill seeds
7,852
—
Seeds with first-build logs
7,841
99.86%
Initial environment success
6,630
84.56%
Environment repair recovery
874
—
Successful environments
7,504
95.70%
Entering task validation
7,446
99.23%
First-pass valid tasks
2,856
38.35%
Task repair recovery
3,222
—
Final validated tasks
6,078
81.63%
Table 6: Selected command- and turn-level statistics for successful teacher trajectories. Shares for command statistics use all 39,136 command occurrences; transition probabilities are normalized within the current turn state.
View
Statistic
Value
Command usage
cat occurrences
18,168 (46.4%)
Top three commands
27,207 (69.5%)
Top ten commands
34,598 (88.4%)
Observation commands
33,140 (84.7%)
Turn dynamics
Observation-only first turn
1,214 (95.6%)
Observation-only → observation-only
4,324 (55.6%)
Action-only → observation-only
1,399 (53.1%)
Action-only → action-only
758 (28.7%)
Table 7: Outcomes of three artifact-generation orders on 100 shared semantic paths. Initial validity is computed over tasks that reach validation, while final yield is computed over all selected paths.
Scheme
Reached validation
Initially valid
Final yield
Forward (Ours)
99
46 (46.5%)
83/100
Reverse
91
22 (24.2%)
63/100
Joint
96
36 (37.5%)
65/100
Table 8: Shared-state task construction and Docker round-trip validation.
Step
Stage
Operation
1
Materialize environment
Generate the Dockerfile, fixtures, dependencies, and initialization scripts from the reconstructed task specification.
2
Build and repair
Build the image with docker build. Build or initialization failures are returned to the environment-repair agent for at most three iterations.
3
Capture shared state
Start a temporary container, inspect the task workspace, and record the realized initial state e0.
4
Generate artifacts
Generate the final instruction, reference solution, and verifier using the same reconstructed specification and shared state e0.
5
Baseline validation
Start a clean container, copy and execute the verifier without running the solution, and require reward 0.
6
Oracle validation
Start another clean container, copy and execute solution/solve.sh, run tests/test.sh, and require reward 1.
7
Repair and revalidate
Classify a failure as an instruction, environment, solution, or verifier defect, apply the corresponding repair, and repeat the full validation procedure for at most five rounds.
8
Accept and clean up
Retain the task only after all validation conditions pass, then stop and remove temporary containers and images.
Table 9: Principal training and evaluation configurations.
Setting
Value
Supervised fine-tuning
Base models
Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B
Training data
1.2K complete successful trajectories
Training strategy
Full-parameter SFT with BF16 and ZeRO-3
Epochs / effective batch size
3 / 64
Learning rate / schedule
1×10−5 / cosine with 0.1 warmup ratio
Maximum sequence length
32,768 tokens
Evaluation
Benchmark / agent
Terminal-Bench 2.1 / Terminus-2
Attempts / timeout
3 per task / 2 hours per attempt
Temperature
1.0
Context length
FACET models: 32,768 tokens; other models: officially supported maximum
Table 10: End-to-end comparison over 500 common skill-pair inputs. Packages denotes complete Harbor task packages, Validated denotes tasks passing oracle validation, and Yield is computed over all inputs. P@1 and P@3 are evaluated on the tasks retained by each pipeline, and Avg. Cmds. is the average number of terminal commands per rollout.
Pipeline
Packages
Validated
Yield
P@1
P@3
Avg. Cmds.
Baseline
437
78
15.6%
80.8%
85.9%
12.8
TW
449
139
27.8%
58.8%
64.0%
17.0
FACET (Ours)
395
350
70.0%
25.1%
33.1%
21.5
Why it matters
Training terminal agents needs large amounts of executable, verifiable tasks that are too costly to hand-write, but if the instruction, environment, solution, and checker don't match, the training data itself becomes broken. FACET shows a structural way to keep these pieces consistent, suggesting that far less training data can still produce meaningful agent improvements.
Terms in this paper
terminal agent · an AI system that operates a command-line interface, e.g. manipulating files or running programs
verifier · executable code that automatically checks whether a task was completed correctly
Harbor format · a standardized folder structure packaging a task's environment, solution, tests, and instructions
P@1 / P@3 · the pass rate when an agent is given one attempt versus three attempts at a task
fine-tuning · further training an existing model on specific data to improve its performance on a task
Original abstract (English)
Training terminal agents requires scalable executable supervision, yet synthesizing high-quality terminal tasks remains challenging. Each task couples an instruction, an initialized environment, a reference solution, and an executable verifier; if these artifacts are generated from inconsistent assumptions, the resulting task may be unsolvable or incorrectly evaluated. Meanwhile, multi-stage synthesis can discard the goals, dependencies, state tran
Authors · Kou Shi, Zun Wang, Qisheng Su, Shiting Huang, Ziao Zhang, Zhen Fang, Qingnan Ren, Jin Liu, Yu Zeng, Yiming Zhao, Lin Che
