매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis

arXiv:2608.185802026-08-20

AI 에이전트에게 '터미널 문제집'을 정합성 있게 만들어주는 파이프라인, FACET

터미널(명령줄) 작업을 수행하는 AI 에이전트를 훈련시키려면 지시문, 초기 환경, 정답 풀이, 채점기(verifier)가 서로 딱 맞아떨어지는 '문제 세트'가 대량으로 필요하다. FACET은 실제 공개된 작업 스킬들을 재조합해 그럴듯한 시나리오를 복원하고, 실행 가능한 도커 환경을 먼저 만든 뒤 그 환경을 기준으로 지시문·정답·채점기를 순서대로 생성해 서로 어긋나지 않게 만든다. 이렇게 만든 6,078개 작업으로 얻은 성공 사례 1,200개만 학습에 써도 Qwen3.5 계열 모델(4B~27B)의 Terminal-Bench 2.1 점수가 모두 눈에 띄게 올랐다.

무엇을 했나

  1. 71,000개 이상의 공개 에이전트 스킬(작업 절차)을 수집·정제한 뒤, 비슷한 목표를 가진 스킬들을 묶어 하나의 그럴듯한 사용자 시나리오로 복원한다.
  2. 시나리오를 목표·맥락·능력·상태·입출력 다섯 관점으로 정리해 정답 풀이 참조안과 지시문 참조안을 만들고, 둘이 같은 초기·최종 상태를 가리키는지 모델로 대조 검증한다.
  3. 실행 환경(도커 이미지, 파일, 서비스)을 먼저 만들고 빌드 오류가 나면 최대 3번까지 자동 수리한 뒤, 그 완성된 환경을 보면서 지시문→정답→채점기 순서로 생성해 셋이 실제 파일·경로·스키마를 공유하게 만든다.
  4. 최종적으로 환경 빌드, 채점기의 초기 상태 거부, 정답 실행, 채점기의 최종 상태 통과 네 조건을 모두 만족해야 문제로 채택되며, 실패 시 문제 전체가 아니라 원인이 된 부분(환경/정답/채점기)만 최대 5번 수리한다.
  5. 생성 순서 비교 실험에서 지시문→정답→채점기 순서(Forward)가 채점기를 먼저 만드는 방식보다 초기 유효율이 훨씬 높았고(46.5% vs 24.2%), 최종적으로 6,078개 검증된 작업과 1,200개 성공 궤적으로 미세조정한 Qwen3.5 모델들은 4B에서 17.60→24.72, 9B에서 27.34→35.58, 27B에서 40.82→47.57로 성능이 올라 15배 큰 모델(397B, 49.06점)에 근접했다.
Figure 1: Overview of FACET. Stage 1 collects skills and constructs the scenario–skill repository. Stage 2 understands the selected skills, explores and recovers a joint scenario, expands it into a complete representation, and builds aligned solution and instruction references. Stage 3 constructs and validates the environment before generating the final task artifacts, with bounded repair loops for failed builds and tests.
Figure 1: Overview of FACET. Stage 1 collects skills and constructs the scenario–skill repository. Stage 2 understands the selected skills, explores and recovers a joint scenario, expands it into a complete representation, and builds aligned solution and instruction references. Stage 3 constructs and validates the environment before generating the final task artifacts, with bounded repair loops for failed builds and tests.
Table 1: Trajectory- and task-level comparison of terminal-agent datasets. Trajectories are collected using the Terminus-2 scaffold, and task performance is evaluated using DeepSeek-V4-Pro with Terminus-2. Turns and Tests denote the average interaction turns per trajectory and executable checkpoints per task. Detailed protocols are provided in Appendix A.6.
DatasetTrajectoryTask
#Traj.Turns#TasksTestsP@1P@3
Nemotron-Terminal (29)5K6.1215K6.1840.6748.00
Endless-Terminals (13)2004.532,4925.5183.0087.00
Terminal-Lego (33)32K5.7715K16.6047.0049.00
TerminalWorld (8)20011.941,5303.9857.0082.00
Tmax (16)50011.1415K3.2980.0086.00
FACET (ours)1.2K11.866K22.7727.0035.00
Figure 2: Analysis of execution and synthesis failures. (a) Distribution of failed or errored verifier checks among unsuccessful teacher rollouts; three rollouts without a parsed FAILED/ERROR result are omitted. (b) Distribution of initial validation failure types under the Forward, Reverse, and Joint generation schemes.
Figure 2: Analysis of execution and synthesis failures. (a) Distribution of failed or errored verifier checks among unsuccessful teacher rollouts; three rollouts without a parsed FAILED/ERROR result are omitted. (b) Distribution of initial validation failure types under the Forward, Reverse, and Joint generation schemes.
Table 2: Results on Terminal-Bench 2.1. Scores under our evaluation setting are averaged over three independent attempts per task.
ModelSizeAgentTerminal-Bench 2.1
Reported reference models
GPT-5.5 (xhigh) (31)—Terminus-278.00
Claude Opus 4.7 (max) (31)—Terminus-266.10
Gemini 3 Pro (high) (31)—Gemini CLI65.80
Intern-S2-Preview-397B (7)397BTerminus-267.42
MiniMax M3 (19)428BTerminus-266.00
GLM-5.1 (max) (31)744BClaude Code58.70
Models evaluated under our setting
Qwen3.6-27B27BTerminus-253.93
Qwen3.5-397B-A17B397BTerminus-249.06
Kimi-K2.61TTerminus-259.93
DeepSeek-V4-Pro-Preview (high)1.6TTerminus-273.03
Qwen3.5 base models
Qwen3.5-4B4BTerminus-217.60
Qwen3.5-9B9BTerminus-227.34
Qwen3.5-27B27BTerminus-240.82
Fine-tuned models
FACET-Terminal-Qwen3.5-4B4BTerminus-224.72 (+7.12)
FACET-Terminal-Qwen3.5-9B9BTerminus-235.58 (+8.24)
FACET-Terminal-Qwen3.5-27B27BTerminus-247.57 (+6.75)
Figure 3: Distributions of source skills and synthesized tasks. (a) The retained skill corpus spans five top-level families and 34 fine-grained categories. (b) The 6,078 validated tasks are distributed across nine task families, whose individual shares range from 9.59% to 11.99%.
Figure 3: Distributions of source skills and synthesized tasks. (a) The retained skill corpus spans five top-level families and 34 fine-grained categories. (b) The 6,078 validated tasks are distributed across nine task families, whose individual shares range from 9.59% to 11.99%.
Table 3: Top-level source-skill distribution.
CategorySkillsShare
AI, agents, and tools15,18221.28%
Software, systems, and security15,05921.11%
Data, analysis, and research12,40917.39%
Documents, productivity, and workflows11,26715.79%
Multimedia, creation, and publishing17,42424.42%
Total71,341100.00%
Figure 4: Strict task-level pass rates for the 20 most frequent skill tags among the 6,066 completed rollouts. Error bars indicate Wilson 95% confidence intervals, and the dashed line denotes the overall pass rate. Tags are selected by frequency and ordered by observed pass rate.
Figure 4: Strict task-level pass rates for the 20 most frequent skill tags among the 6,066 completed rollouts. Error bars indicate Wilson 95% confidence intervals, and the dashed line denotes the overall pass rate. Tags are selected by frequency and ordered by observed pass rate.
Table 4: Fine-grained source-skill categories. “Global” is the share of all 71,341 skills; “within parent” is the share inside the corresponding top-level category.
ParentFine-grained categoryCountGlobalWithin parent
AI, agents, and toolsAgent orchestration and automation2,7823.90%18.32%
Prompt and model calls3,3104.64%21.80%
Skills, plugins, and extensions1,6372.29%10.78%
MCP and external tools2,5793.62%16.99%
Memory, RAG, and knowledge bases3,4394.82%22.65%
Multi-agent collaboration1,4352.01%9.45%
Software, systems, and securityCode generation and development3,0654.30%20.35%
Testing, debugging, and code quality2,1673.04%14.39%
Git, build, and dependency management1,9042.67%12.64%
Deployment, containers, and DevOps2,0692.90%13.74%
System administration and CLI3,5765.01%23.75%
Security, privacy, and compliance2,2783.19%15.13%
Data, analysis, and researchJSON, YAML, and XML2,7523.86%22.18%
CSV, Excel, and spreadsheets1,5442.16%12.44%
Databases and SQL1,6272.28%13.11%
Data cleaning, conversion, and validation1,5952.24%12.85%
Statistical analysis and metrics1,8922.65%15.25%
Visualization and dashboards1,4191.99%11.44%
Search, research, and extraction1,5802.21%12.73%
Documents, productivity, and workflowsDocuments and Markdown1,4962.10%13.28%
Reports, summaries, and briefs1,8652.61%16.55%
PDF, Office, and presentations1,4292.00%12.68%
Office and personal productivity1,5922.23%14.13%
Project, task, and schedule management1,6822.36%14.93%
System integration and automation1,5972.24%14.17%
Audit, checklists, and operation records1,6062.25%14.25%
Multimedia, creation, and publishingImage generation and editing2,6993.78%15.49%
Design, drawing, and visual assets1,8582.60%10.66%
Audio, speech, and music1,9172.69%11.00%
Video, animation, and captions2,4913.49%14.30%
Figure 5: Command-level patterns in successful teacher trajectories. The left panel shows the most frequent shell commands, colored by their dominant contextual class. The right panel reports row-normalized transitions between adjacent assistant-turn states.
Figure 5: Command-level patterns in successful teacher trajectories. The left panel shows the most frequent shell commands, colored by their dominant contextual class. The right panel reports row-normalized transitions between adjacent assistant-turn states.
Table 5: Task-construction funnel. Percentages in the last column use the immediately preceding comparable stage.
StageCountStage retention
Scenario–skill seeds7,852—
Seeds with first-build logs7,84199.86%
Initial environment success6,63084.56%
Environment repair recovery874—
Successful environments7,50495.70%
Entering task validation7,44699.23%
First-pass valid tasks2,85638.35%
Task repair recovery3,222—
Final validated tasks6,07881.63%
Table 6: Selected command- and turn-level statistics for successful teacher trajectories. Shares for command statistics use all 39,136 command occurrences; transition probabilities are normalized within the current turn state.
ViewStatisticValue
Command usagecat occurrences18,168 (46.4%)
Top three commands27,207 (69.5%)
Top ten commands34,598 (88.4%)
Observation commands33,140 (84.7%)
Turn dynamicsObservation-only first turn1,214 (95.6%)
Observation-only → observation-only4,324 (55.6%)
Action-only → observation-only1,399 (53.1%)
Action-only → action-only758 (28.7%)
Table 7: Outcomes of three artifact-generation orders on 100 shared semantic paths. Initial validity is computed over tasks that reach validation, while final yield is computed over all selected paths.
SchemeReached validationInitially validFinal yield
Forward (Ours)9946 (46.5%)83/100
Reverse9122 (24.2%)63/100
Joint9636 (37.5%)65/100
Table 8: Shared-state task construction and Docker round-trip validation.
StepStageOperation
1Materialize environmentGenerate the Dockerfile, fixtures, dependencies, and initialization scripts from the reconstructed task specification.
2Build and repairBuild the image with docker build. Build or initialization failures are returned to the environment-repair agent for at most three iterations.
3Capture shared stateStart a temporary container, inspect the task workspace, and record the realized initial state e0.
4Generate artifactsGenerate the final instruction, reference solution, and verifier using the same reconstructed specification and shared state e0.
5Baseline validationStart a clean container, copy and execute the verifier without running the solution, and require reward 0.
6Oracle validationStart another clean container, copy and execute solution/solve.sh, run tests/test.sh, and require reward 1.
7Repair and revalidateClassify a failure as an instruction, environment, solution, or verifier defect, apply the corresponding repair, and repeat the full validation procedure for at most five rounds.
8Accept and clean upRetain the task only after all validation conditions pass, then stop and remove temporary containers and images.
Table 9: Principal training and evaluation configurations.
SettingValue
Supervised fine-tuning
Base modelsQwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B
Training data1.2K complete successful trajectories
Training strategyFull-parameter SFT with BF16 and ZeRO-3
Epochs / effective batch size3 / 64
Learning rate / schedule1×10−5 / cosine with 0.1 warmup ratio
Maximum sequence length32,768 tokens
Evaluation
Benchmark / agentTerminal-Bench 2.1 / Terminus-2
Attempts / timeout3 per task / 2 hours per attempt
Temperature1.0
Context lengthFACET models: 32,768 tokens; other models: officially supported maximum
Table 10: End-to-end comparison over 500 common skill-pair inputs. Packages denotes complete Harbor task packages, Validated denotes tasks passing oracle validation, and Yield is computed over all inputs. P@1 and P@3 are evaluated on the tasks retained by each pipeline, and Avg. Cmds. is the average number of terminal commands per rollout.
PipelinePackagesValidatedYieldP@1P@3Avg. Cmds.
Baseline4377815.6%80.8%85.9%12.8
TW44913927.8%58.8%64.0%17.0
FACET (Ours)39535070.0%25.1%33.1%21.5

왜 중요한가

터미널 에이전트를 학습시키려면 사람이 일일이 만들기 힘든 '실행 가능하고 검증 가능한' 문제가 대량으로 필요한데, 지시문·환경·정답·채점기가 서로 어긋나면 학습 데이터 자체가 오염된다. FACET은 이런 정합성 문제를 구조적으로 해결하는 방법을 보여줘, 적은 데이터로도 효율적인 에이전트 훈련이 가능함을 시사한다.

이 논문의 용어

  • 터미널 에이전트 · 명령줄(터미널)에서 파일 조작, 프로그램 설치, 명령어 실행 등을 수행하는 AI
  • 검증기(verifier) · 작업이 올바르게 완료됐는지 자동으로 확인하는 실행 가능한 채점 코드
  • Harbor 포맷 · 환경/정답/테스트/지시문을 표준화된 폴더 구조로 담는 작업 패키지 형식
  • P@1 / P@3 · 한 번 또는 세 번 시도했을 때 문제를 통과하는 비율
  • 미세조정(fine-tuning) · 이미 학습된 모델을 특정 데이터로 추가 학습시켜 성능을 개선하는 과정

논문 원문 초록 (영문)

Training terminal agents requires scalable executable supervision, yet synthesizing high-quality terminal tasks remains challenging. Each task couples an instruction, an initialized environment, a reference solution, and an executable verifier; if these artifacts are generated from inconsistent assumptions, the resulting task may be unsolvable or incorrectly evaluated. Meanwhile, multi-stage synthesis can discard the goals, dependencies, state tran

저자 · Kou Shi, Zun Wang, Qisheng Su, Shiting Huang, Ziao Zhang, Zhen Fang, Qingnan Ren, Jin Liu, Yu Zeng, Yiming Zhao, Lin Che

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Kou Shi et al., arXiv:2608.18580, CC BY 4.0