QuoteBench: How Matched Scores Can Hide Command-Path Failures

arXiv:2608.135472026-08-12

AI 코딩 에이전트의 '정답률'은 명령어가 실행 직전에 다시 파싱될 때 무너진다

AI 코딩 에이전트가 생성한 Bash 명령어는 실행되기 전에 한 번 더 감싸지거나 재해석되는 경우가 많은데, 기존 벤치마크는 이 지점에서 벌어지는 손상과 모델 자체의 생성 오류를 구분하지 못했다. QuoteBench는 56개 과제로 이 둘을 실험적으로 분리해, 명령어가 큰따옴표 안에 다시 삽입되는 상황(중첩 전송)에서 성공률이 55.4~73.2점(퍼센트 포인트) 떨어지고, 이 위험을 모델에 미리 알려주면 6개 설정에서 30.4~60.7점이 회복됨을 보였다. GPT-5.6-sol처럼 겉보기 점수 변화는 -3.6점에 불과하지만 실제로는 -64.3점의 손상과 +60.7점의 보완이 서로 상쇄된 경우도 발견됐다.

무엇을 했나

문제의식: LLM 에이전트가 만든 Bash 명령어는 원격 접속(ssh), 컨테이너 실행, CI 파이프라인 등을 거치며 큰따옴표 안에 다시 끼워 넣어지는데, 이 과정에서 따옴표나 특수문자가 깨져도 기존 평가 방식(정답 여부만 보는 매칭 점수)으로는 원인을 알 수 없었다.
방법: 저자는 14개 실제 장애 사례에서 뽑은 14개 작업군, 56개 과제로 QuoteBench를 만들고, 모델이 명령어를 만드는 방식(생성 계약)과 그 명령어가 실행되는 경로(실행 전송)를 독립적으로 교차시켰다. 같은 모델 응답을 원본 그대로 실행할 때와, 일부러 이스케이프 처리를 하지 않은 파서를 하나 추가해 재실행할 때를 비교해 손상분을 측정하고, 모델에게 '네 답은 큰따옴표 안에 다시 넣어질 것'이라고 미리 알려주는 조건(공개된 경계)을 추가해 회복분을 측정했다.
결과: 원본 그대로 실행하면 최상위 모델들은 이미 91.1~100%로 거의 포화 상태였지만, 재파싱 과정을 하나 추가하는 것만으로 성공률이 55.4~73.2점 떨어졌다. 이 위험을 미리 고지하면 8개 설정 중 6개에서 30.4~60.7점이 회복됐고, 나머지 2개는 거의 변화가 없거나 소폭 나빠졌다.
의미 있는 발견: 겉보기 점수(매칭 점수)만 보면 GPT-5.6-sol은 거의 변화가 없어 보이지만(-3.6점), 실제로는 큰 손상(-64.3점)과 큰 회복(+60.7점)이 우연히 상쇄된 결과였다. 또한 실행 경로를 바꾸는 것만으로 26개 모델 비교 쌍 중 최소 1개는 순위가 명백히 뒤집혔다.
실용적 시사점: 올바른 이스케이프 처리나 임시 스크립트 사용 같은 간단한 해법으로 이 문제를 완전히 없앨 수 있지만, 실제 에이전트 시스템 6곳을 조사한 결과 이런 경계 처리가 모델이 아니라 그 뒤 단계(하네스)에서 이뤄지고 있어 모델 혼자서는 통제할 수 없는 경우가 많았다.

Figure 1: Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract-then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The RR and NN cells are matched. The RN cell measures fixed-reply damage, and NN−RN is the realized contract-conditioned contrast. Panel (b) works the decomposition for GPT-5.6-sol.

Table 1: One worked QuoteBench task (write-file, tier 3). The ⟨⟨⋅⟩⟩ markers delimit the exact literal payload and are not part of it. The ↩ symbol marks a newline that the payload itself contains. The validator compares exact final bytes, so any command that reaches the target state passes. Appendix Table 7 maps the full family set to the covered mechanisms. The artifact contains all 56 instructions, fixtures, oracles, and validators.
Instruction	Create a file named out.txt whose content is exactly the text ⟨⟨back\slash and `date` here↩ ’single’ "double" $(pwd) 100%⟩⟩ followed by a single trailing newline.
Hazards	backslash, backtick, command substitution, multiline, mixed quotes
Naive command	echo "back\slash and `date` here …" > out.txt; the backticks execute date, $(pwd) expands locally, and the payload’s inner double quote terminates the string early
Final-state gate	exact bytes of out.txt, including one trailing newline, with no collateral files
One passing command	a single-quoted printf with an embedded-quote splice (the machine oracle). Any other correct encoding receives equal credit

Figure 2: Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix Table 13 lists the queried effort settings and request parameters. Table 2 separately summarizes the best observed setting per base model. Columns follow the mechanism groups in Appendix Table 7. SSH-like tasks use the local two-shell simulation. Rows are ordered by matched-nested score in this frozen sweep, which differs from the best-observed order in Table 2.

Table 2: Best-observed QuoteBench scorecard. Each base model contributes the measured setting with the highest matched-nested All-56 score. Ties prefer default, then lower mean provider-reported output tokens. default means that the request omitted the effort field. The number and names of measured settings differ by provider and appear in Figure 4 and Appendix B. This table is descriptive: each cell is one stored trial-0 generation and the selection is a within-model maximum over rungs; Appendix E.1.6 reports draw-to-draw spread. The Qwen rows expose only a think toggle rather than an effort ladder, so their best-observed setting is taken from the same-window sweep; for Qwen3.5-27B, the only Qwen size with a ladder row in Table 12, the resulting one-task difference is serving-window drift. Fixed same-window configurations support the mechanism analysis in Table 4.
Model	Best observed setting	Control	Hostile	All 56 (%)	Hostile LOFO range (%)
GPT-5.5	xhigh	14/14	42/42	100.0	[100.0, 100.0]
Opus-5	xhigh	14/14	42/42	100.0	[100.0, 100.0]
Fable-5	max	14/14	42/42	100.0	[100.0, 100.0]
GPT-5.6-sol	high	14/14	41/42	98.2	[97.4, 100.0]
Opus-4.8	max	12/14	41/42	94.6	[97.4, 100.0]
Gemini-3.1-Pro	low	14/14	37/42	91.1	[87.2, 89.7]
Sonnet-4.6	high	9/14	27/42	64.3	[61.5, 69.2]
Gemini-3.5-Flash	medium	10/14	26/42	64.3	[59.0, 66.7]
Haiku-4.5	medium	8/14	13/42	37.5	[28.2, 33.3]
Qwen3.5-27B	non-think	8/14	9/42	30.4	[15.4, 23.1]
Qwen3.5-4B	think	5/14	7/42	21.4	[10.3, 17.9]
Qwen3.5-9B	non-think	5/14	5/42	17.9	[5.1, 12.8]
Gemini-3.1-Flash-Lite	default	6/14	2/42	14.3	[2.6, 5.1]

Figure 3: Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The RR value is raw success, RN is the nested-replay pass rate of raw-conditioned replies, and NN is matched nested success. The NN−RN gap is compensation. Appendix Table 14 reports every rung, including the byte-identical Gemini-3.1-Flash-Lite settings.

Table 3: Absolute GNU replay pass rates in the separate Study-B campaign, pooled over each model’s measured effort ladder and three trials per cell.
Model	Attempts	Raw (%)	Native (%)	Δ (pp)
Opus-4.8	840	95.4	98.0	+2.6
Opus-5	840	98.2	97.4	−0.8
Fable-5	840	99.3	97.1	−2.1
Gemini-3.1-Pro	504	98.8	95.0	−3.8
GPT-5.6-sol	672	96.9	94.3	−2.5
Gemini-3.5-Flash	672	95.7	85.7	−10.0

Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched-nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order. Backward or dominated segments show measured non-monotonicity. Panels group model families for legibility, and Qwen sizes contribute two-point think-toggle trajectories.

Table 4: All four crossover cells for the eight same-window configurations. Cell notation follows Figure 1: generation contract precedes transport. Cells are pass rates (%). Effects are percentage points. Damage is RN−RR, compensation is NN−RN, and the matched gap is NN−RR. Appendix Table 14 reports the interaction at every measured rung, and Table 15 gives the sensitivity tests.
Model	RR	RN	NR	NN	Damage	Comp.	Matched gap
GPT-5.6-sol	94.6	30.4	55.4	91.1	−64.3	+60.7	−3.6
GPT-5.5	100.0	28.6	50.0	89.3	−71.4	+60.7	−10.7
Opus-5	96.4	30.4	42.9	89.3	−66.1	+58.9	−7.1
Gemini-3.1-Pro	98.2	25.0	33.9	80.4	−73.2	+55.4	−17.9
Gemini-3.5-Flash	96.4	28.6	67.9	58.9	−67.9	+30.4	−37.5
Opus-4.8	91.1	26.8	62.5	57.1	−64.3	+30.4	−33.9
Qwen3.5-27B	85.7	30.4	83.9	30.4	−55.4	0.0	−55.4
Gemini-3.1-Flash-Lite	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3

Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered scale. Blank cells mark rungs the provider does not expose. Models follow descending GNU replay campaign effect. Labels are not comparable compute budgets across providers.

Table 5: Private-v2 crossover on 42 hostile payloads under the single-clause disclosed-boundary contract. The private set is hostile-only and not difficulty-matched to the public core, so absolute rates are interpreted within this set. Cells are pass rates. Effects are percentage points. Damage is RN−RR, compensation is NN−RN, interaction is (NN−NR)−(RN−RR), and the matched gap is NN−RR.
Model	RR	RN	NR	NN	Damage	Comp.	Interaction	Matched gap
GPT-5.6-sol	92.9	19.0	50.0	97.6	−73.8	+78.6	+121.4	+4.8
Opus-4.8	92.9	16.7	59.5	42.9	−76.2	+26.2	+59.5	−50.0

Table 6: Private-v1 fixed raw replies under three transports on 42 tasks. This campaign predates the private-v2 crossover in Table 5. The two tables therefore use different generations. The first three columns report tasks passed out of 42. Script gain is the temporary-script rate minus the nested-wrapper rate, in percentage points.
Model	Raw	Nested wrapper	Temporary script	Script gain (pp)
GPT-5.6-sol	41/42	8/42	41/42	+78.6
Opus-4.8	40/42	7/42	40/42	+78.6
Qwen3.5-27B	30/42	9/42	30/42	+50.0

Table 7: Mechanism groups used to construct the frozen core; the mapping documents coverage of the surveyed failure mechanisms. Benchmark analyses weight the 14 operation families equally.
Mechanism group	Representative failures	QuoteBench families
Literal quote and expansion	Apostrophes, double quotes, dollars, backticks, multiline payloads	write-file, JSON writing, Git commit, environment passing, heredoc writing
Word splitting and path semantics	Spaces, globs, leading dashes, hostile filenames, argument boundaries	argv passing, hostile filenames, find/glob, bulk rename
Embedded-language escaping	Regex versus literal matching, sed replacement, AWK string processing	grep count, sed replace, field lookup, JSON writing
Second parser or remote-like expansion	Local expansion before a second shell, argument joining, heredoc transport	SSH-like nested execution, SSH-like heredoc
Command-boundary representation	Command string, shell stdin, temporary file, argv, provider tool schema	raw/nested crossover, native-tool study, script bypass, typed pilot

Table 8: Command boundaries in six public agent systems, inspected at fixed commits. Contract is what the system asks the model to produce. A nested boundary arises on a separate axis from what the command targets. Observed boundary is what the harness then does with the reply. The classification concerns the parser boundary only. Sources: 20; 27; 13; 28; 22; 18.
System	Contract	Source anchor	Observed boundary
Codex	native	core/src/shell.rs:20--30, commit fa1d4c4	command string becomes shell -c/-lc R
SWE-agent	raw	agents.py:936--967 and swe_env.py:197--222, commit 3ea751c	agent action enters a persistent Bash session
LangChain	native	shell_tool.py:217--232,491--515, commit b3a6d9a	structured command: string is written to shell stdin
Terminal-Bench	raw	tmux_session.py:26--33,75--173, commit d28711d	command/key strings enter an interactive shell through tmux
OpenHands	native	acp-command.ts:11--45,65--147, commit 850bd64	human-readable command is tokenized to argv, and spawn has no shell
AutoGen	native	_docker_code_executor.py:327--363, commit 027ecf0	generated code is written to a temporary file and invoked by argv

Table 9: Real-ssh grounding. Each stored raw reply is replayed through bash -c and through a real ssh localhost "R" remote command; ssh damage is the second minus the first. It matches the synthetic nested damage to the decimal for seven of eight configurations; the disclosed-reply cells are grounded in Appendix Table 21. Zero model calls.
Configuration	Raw	ssh	ssh damage	Nested damage
GPT-5.6-sol	94.6	30.4	−64.3	−64.3
GPT-5.5	100.0	28.6	−71.4	−71.4
Opus-5	96.4	30.4	−66.1	−66.1
Gemini-3.1-Pro	98.2	26.8	−71.4	−73.2
Gemini-3.5-Flash	96.4	28.6	−67.9	−67.9
Opus-4.8	91.1	26.8	−64.3	−64.3
Qwen3.5-27B	85.7	30.4	−55.4	−55.4
Gemini-3.1-Flash-Lite	78.6	19.6	−58.9	−58.9

Table 10: Campaign map. All replays are zero-call executions of stored replies in the pinned container.
Campaign	Generations	Design	Feeds
Study A same-window sweep	8 configs × 56 × 2 contracts	one randomized window, effort unset	Tables 4, 15
Study A ladder sweep	44 rungs, 11 configs	per-provider windows	Tables 2, 12
Study A rung crossover	30 rungs, 7 configs, 26 crossover pts	replay both transports	Table 14, Fig. 3
Public three-draw repeat	8 configs × 56 × 2 × 2 draws	same design as the sweep	Appendix E.1.6
Study B native tool	8,736 generations, 17,472 replays	observational, both userlands	Tables 3, 17, 18
Private-v2 crossover	2 models × 42 hostile payloads	one serving window	Table 5
Private-v1 replay	3 models × 42 tasks	earlier generations; script bypass	Table 6
BSD-live legacy	6 configurations	BSD-elicited, GNU-replayed	Table 16
Real-ssh grounding	8 configs × 56	ssh localhost replay	Table 9
Advice arm	8 configs × 56 × 2	same-window paired advice contrast	Table 19
Grammar crossover	8 configs × 56 × 2 disclosed	replay-only wrapper 2×2	Table 20
Real-ssh full crossover	6 configs × 56 × 2	disclosed replies on real ssh	Table 21
JSON boundary	6 configs × 56	serializer replay	Table 22

Table 11: Calibration of the unset-effort sweep arm against each configuration’s labelled ladder. Distance is unset success minus the lowest-rung success, in percentage points. The two measurements come from different serving windows, so small differences are descriptive only.
Model	Unset (%)	Nearest rung	Distance to lowest rung
GPT-5.6-sol	91.1	low	0.0
GPT-5.5	89.3	high	−5.4
Opus-5	89.3	medium	+28.6
Gemini-3.1-Pro	80.4	none within the ladder	−10.7
Gemini-3.5-Flash	58.9	low	+1.8
Opus-4.8	57.1	xhigh	+17.9
Gemini-3.1-Flash-Lite	14.3	all four rungs tie	0.0

Table 12: Matched-nested effort ladders under the disclosed-boundary contract. Within each row, success rates and mean provider-reported output tokens follow the setting order in the second column. Each point uses trial 0 over 56 tasks.
Model	Settings (in order)	Success (%)	Mean output tokens
GPT-5.6-sol	low / medium / high / xhigh	91.1 / 92.9 / 98.2 / 98.2	362 / 565 / 773 / 1,073
GPT-5.5	low / medium / high / xhigh	94.6 / 92.9 / 91.1 / 100.0	507 / 655 / 1,164 / 2,757
Opus-5	low / medium / high / xhigh / max	60.7 / 91.1 / 96.4 / 100.0 / 98.2	796 / 1,336 / 1,458 / 2,421 / 3,499
Fable-5	low / medium / high / xhigh / max	71.4 / 75.0 / 96.4 / 92.9 / 100.0	332 / 569 / 843 / 1,212 / 2,396
Opus-4.8	low / medium / high / xhigh / max	39.3 / 48.2 / 50.0 / 62.5 / 94.6	835 / 1,291 / 1,706 / 3,481 / 6,960
Gemini-3.1-Pro	low / medium / high	91.1 / 91.1 / 91.1	5,267 / 5,753 / 6,308
Sonnet-4.6	low / medium / high / xhigh	23.2 / 28.6 / 64.3 / 62.5	28 / 1,411 / 4,337 / 4,539
Gemini-3.5-Flash	minimal / low / medium / high	57.1 / 58.9 / 64.3 / 62.5	3,851 / 3,108 / 3,081 / 3,389
Haiku-4.5	low / medium / high / xhigh	32.1 / 37.5 / 32.1 / 26.8	5,432 / 4,717 / 4,708 / 4,783
Qwen3.5-27B	non-thinking / thinking	28.6 / 28.6	20 / 7,489
Gemini-3.1-Flash-Lite	minimal / low / medium / high	14.3 / 14.3 / 14.3 / 14.3	20 / 20 / 20 / 20

Table 13: Study-A model identifiers and request parameters. The effort column lists exactly the settings queried; sweep arms omitted the effort field. Temperature is reported only where the interface accepts it.
Display name	Model identifier	Effort settings queried	Decoding parameters
GPT-5.6-sol	gpt-5.6-sol	low, medium, high, xhigh	max output tokens 16,000, temperature not sent
GPT-5.5	gpt-5.5	low, medium, high, xhigh	max output tokens 16,000, temperature not sent
Opus-5	claude-opus-5	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Opus-4.8	claude-opus-4-8	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Fable-5	claude-fable-5	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Sonnet-4.6	claude-sonnet-4-6	low, medium, high, xhigh	provider defaults, no sampling or length control sent
Haiku-4.5	claude-haiku-4-5	low, medium, high, xhigh	provider defaults, no sampling or length control sent
Gemini-3.1-Pro	gemini-3.1-pro-preview	low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.5-Flash	gemini-3.5-flash	minimal, low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.1-Flash-Lite	gemini-3.1-flash-lite-preview	minimal, low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Qwen3.5-27B	Qwen/Qwen3.5-27B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-9B	Qwen/Qwen3.5-9B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-4B	Qwen/Qwen3.5-4B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking

Table 14: Crossover at every measured rung. Cells are percentages and effects are percentage points, defined as in Table 4. An asterisk marks the descriptive masked-fragility rule. Gemini-3.1-Flash-Lite returns byte-identical replies at all four settings and is retained only to document that the provider did not expose a usable ladder.
Configuration	Rung	RR	RN	NR	NN	Damage	Compensation	Matched gap
GPT-5.6-sol	low	98.2	28.6	53.6	91.1	−69.6	+62.5	−7.1
GPT-5.6-sol	medium	94.6	30.4	48.2	92.9	−64.3	+62.5	−1.8∗
GPT-5.6-sol	high	98.2	32.1	48.2	98.2	−66.1	+66.1	+0.0∗
GPT-5.6-sol	xhigh	98.2	30.4	51.8	98.2	−67.9	+67.9	+0.0∗
Opus-5	low	98.2	28.6	57.1	60.7	−69.6	+32.1	−37.5
Opus-5	medium	100.0	32.1	41.1	91.1	−67.9	+58.9	−8.9
Opus-5	high	96.4	30.4	42.9	96.4	−66.1	+66.1	+0.0∗
Opus-5	xhigh	98.2	32.1	42.9	100.0	−66.1	+67.9	+1.8∗
Opus-5	max	100.0	32.1	46.4	98.2	−67.9	+66.1	−1.8∗
Gemini-3.1-Pro	low	100.0	25.0	32.1	91.1	−75.0	+66.1	−8.9
Gemini-3.1-Pro	medium	98.2	26.8	41.1	91.1	−71.4	+64.3	−7.1
Gemini-3.1-Pro	high	98.2	26.8	35.7	91.1	−71.4	+64.3	−7.1
Fable-5	low	100.0	30.4	48.2	71.4	−69.6	+41.1	−28.6
Fable-5	medium	100.0	30.4	48.2	75.0	−69.6	+44.6	−25.0
Fable-5	high	98.2	32.1	42.9	96.4	−66.1	+64.3	−1.8∗
Fable-5	xhigh	100.0	33.9	46.4	92.9	−66.1	+58.9	−7.1
Fable-5	max	100.0	30.4	35.7	100.0	−69.6	+69.6	+0.0∗
Gemini-3.5-Flash	minimal	96.4	23.2	75.0	57.1	−73.2	+33.9	−39.3
Gemini-3.5-Flash	low	92.9	25.0	67.9	58.9	−67.9	+33.9	−33.9
Gemini-3.5-Flash	medium	96.4	23.2	76.8	64.3	−73.2	+41.1	−32.1
Gemini-3.5-Flash	high	96.4	23.2	71.4	62.5	−73.2	+39.3	−33.9
Opus-4.8	low	87.5	28.6	69.6	39.3	−58.9	+10.7	−48.2
Opus-4.8	medium	92.9	28.6	67.9	48.2	−64.3	+19.6	−44.6
Opus-4.8	high	96.4	33.9	64.3	50.0	−62.5	+16.1	−46.4
Opus-4.8	xhigh	98.2	32.1	60.7	62.5	−66.1	+30.4	−35.7
Opus-4.8	max	98.2	30.4	44.6	94.6	−67.9	+64.3	−3.6∗
Gemini-3.1-Flash-Lite	minimal	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	low	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	medium	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	high	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3

Table 15: Enumerated and Holm-adjusted two-sided family-sign p values for the two primary Study-A components under the single-clause disclosed-boundary contract. Effect sizes are percentage points.
	Fixed-reply transport	Contract-conditioned compensation
Model	Effect [95% CI]	Enum. p	Holm p	Effect [95% CI]	Enum. p	Holm p
GPT-5.6-sol	−64.3 [−80.4,−46.4]	.000244	.001465	+60.7 [+46.4,+75.0]	.000244	.001953
GPT-5.5	−71.4 [−85.7,−55.4]	.000244	.001465	+60.7 [+44.6,+75.0]	.000244	.001953
Opus-5	−66.1 [−82.1,−50.0]	.000244	.001465	+58.9 [+41.1,+75.0]	.000488	.002930
Gemini-3.1-Pro	−73.2 [−87.5,−58.9]	.000122	.000977	+55.4 [+33.9,+73.2]	.001221	.006104
Gemini-3.5-Flash	−67.9 [−82.1,−51.8]	.000244	.001465	+30.4 [+12.5,+48.2]	.013672	.041016
Opus-4.8	−64.3 [−80.4,−46.4]	.000244	.001465	+30.4 [+14.3,+48.2]	.003906	.015625
Qwen3.5-27B	−55.4 [−69.6,−41.1]	.000244	.001465	0.0 [0.0,0.0]	1.000000	1.000000
Gemini-3.1-Flash-Lite	−58.9 [−71.4,−48.2]	.000122	.000977	−5.4 [−10.7,0.0]	.250000	.500000

Table 16: Cross-userland transfer in the earlier BSD-live campaign. Each cell reports pass rate on BSD live execution and GNU replay of the identical stored command. Arms retain their original campaign multiplicity: raw cells contain 56 records, while nested cells contain 56 or 168 depending on the configuration. We interpret only the within-arm BSD→GNU change. The final column summarizes the higher-transfer userland; it is not a counterfactual GNU-targeted generation result.
Model	Raw BSD→GNU	Nested BSD→GNU	Transfers better to
Fable-5	→91.1	→87.5	BSD
Qwen3.5-27B (non-thinking)	→87.5	→32.1	GNU
Qwen3.5-27B (thinking)	→82.1	→50.0	GNU
Gemini-3.1-Pro	→100.0	→96.4	GNU
Gemini-3.5-Flash	→96.4	→67.9	BSD
Opus-4.8	→87.5	→73.8	BSD (raw); tie (nested)

Table 17: Exploratory native-minus-raw effects over each measured effort ladder, ordered by GNU replay effect.
	BSD live	GNU replay
Model	Δ	LOFO	PF/FP	Δ	LOFO	PF/FP
Opus-4.8	+3.10	[+2.18,+3.46]	5/31	+2.62	[+1.67,+2.95]	11/33
Opus-5	−0.36	[−0.51,0.00]	12/9	−0.83	[−1.03,−0.51]	16/9
Fable-5	+0.95	[0.00,+1.15]	3/11	−2.14	[−2.44,−0.64]	22/4
GPT-5.6-sol	−1.19	[−1.92,−0.48]	25/17	−2.53	[−3.37,−0.64]	26/9
Gemini-3.1-Pro	+1.98	[−0.64,+3.21]	8/18	−3.77	[−4.91,−0.64]	23/4
Gemini-3.5-Flash	−4.02	[−4.33,−3.04]	37/10	−9.97	[−10.74,−6.89]	77/10

Table 18: Study-B execution outcomes, aggregated across six models. Each row partitions 4,368 executions. Adherence denotes an invalid one-call tool invocation. Syntax includes parser and command-usage errors. Exit-0 wrong is a silent final-state failure.
Userland	Contract	Pass	Adherence	Syntax	Nonzero	Exit-0 wrong
BSD	raw	4230	0	31	59	48
BSD	native	4236	20	9	41	62
GNU	raw	4252	0	39	36	41
GNU	native	4146	20	22	128	52

Table 19: Matched nested success (of 56) under the disclosed contract without and with escaping advice, paired within one serving window. Advice barely moves the top of the ladder, moves the middle of the ladder the most, and does not reach the bottom.
Configuration	Advice-free	With advice	Δ (pp)
GPT-5.6-sol	54	49	−8.9
GPT-5.5	49	53	+7.1
Opus-5	52	54	+3.6
Sonnet-4.6	32	46	+25.0
Haiku-4.5	25	32	+12.5
Opus-4.8	30	34	+7.1
Qwen3.5-27B	17	18	+1.8
Gemini-3.1-Flash-Lite	7	8	+1.8

Table 20: Grammar crossover (tasks passed of 56). Rows are the disclosed grammar, columns the executed wrapper; the diagonal is matched, the anti-diagonal mismatched. The diagonal advantage separates the same top, middle, and bottom groups as the matched-nested scores.
	Double disclosed	Single disclosed	Diag−anti
Configuration	on double	on single	on single	on double	(pp)
GPT-5.6-sol	54	7	53	10	+80.4
GPT-5.5	49	8	54	8	+77.7
Opus-5	52	8	45	16	+65.2
Sonnet-4.6	32	11	23	16	+25.0
Opus-4.8	30	11	23	18	+21.4
Haiku-4.5	25	13	20	11	+18.8
Qwen3.5-27B	17	11	11	17	+0.0
Gemini-3.1-Flash-Lite	7	19	9	19	−19.6

Table 21: Real-ssh full crossover (tasks passed of 56). Damage is RN−RR and compensation is NN−RN, both on the real ssh path. The final column repeats the synthetic nested compensation for comparison.
Configuration	RR	RN	NR	NN	ssh comp.	Nested comp.
GPT-5.6-sol	53	17	31	51	+60.7	+60.7
GPT-5.5	56	16	28	50	+60.7	+60.7
Opus-5	54	17	24	50	+58.9	+58.9
Opus-4.8	51	15	35	32	+30.4	+30.4
Qwen3.5-27B	48	17	47	17	+0.0	+0.0
Gemini-3.1-Flash-Lite	44	11	45	10	−1.8	−5.4

Table 22: JSON serializer boundary (tasks passed of 56). Correct-serializer damage is ≈0; naive-embedding damage is comparable to the shell nested transport. Unparseable counts replies whose naive JSON embedding fails to parse.
Configuration	Raw	Correct	Naive	Naive damage
GPT-5.6-sol	53	53	21	−57.1
GPT-5.5	56	56	19	−66.1
Opus-5	54	54	22	−57.1
Opus-4.8	51	51	18	−58.9
Qwen3.5-27B	48	48	19	−51.8
Gemini-3.1-Flash-Lite	44	44	8	−64.3

왜 중요한가

에이전트가 실제로 서버에 접속하거나 컨테이너를 다루는 배포 환경에서는 이 논문이 지적한 '재파싱 경계'가 항상 존재하며, 이를 무시한 벤치마크 순위는 실제 배포 성능과 다를 수 있다. 코딩 에이전트를 도입하려는 개발자나 평가자는 모델 점수 하나만 보지 말고 어떤 실행 경로와 계약 조건에서 측정됐는지 함께 확인해야 한다.

이 논문의 용어

매칭 점수(matched score) · 모델이 만든 명령어를 곧바로 실행해 얻는, 일반적으로 보고되는 성공률
생성 계약(generation contract) · 모델에게 명령어를 어떤 형식·전제로 만들라고 지시하는 방식
실행 전송(execution transport) · 모델이 만든 명령어가 실제 셸에 도달하기까지 거치는 경로, 예를 들어 그대로 실행되거나 다시 따옴표로 감싸져 실행되는 방식
중첩 전송(nested transport) · 명령어가 ssh나 컨테이너 실행처럼 큰따옴표 안에 한 번 더 삽입되어 재해석되는 실행 경로
최종 상태 검증(final-state validation) · 명령어 실행 후 파일, 인자, 깃 기록 등 결과 상태가 정확히 맞는지만 확인하는 채점 방식

논문 원문 초록 (영문)

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

저자 · Shangao Li

arXiv에서 원문 보기

METAL LAB 최신 기사

그림 출처: Shangao Li et al., arXiv:2608.13547, CC BY 4.0

QuoteBench: How Matched Scores Can Hide Command-Path Failures

무엇을 했나

왜 중요한가

이 논문의 용어

최신 논문

METAL LAB 최신 기사