QuoteBench: How Matched Scores Can Hide Command-Path Failures

arXiv:2608.135472026-08-12

AI编程智能体的评测分数可能掩盖了命令在被二次解析时就已经失效的事实

LLM编程智能体生成的Bash命令,常常要先被序列化、包裹,再被下游环节重新解析后才真正执行,而传统只看命令是否执行成功的评测分数,分不清失败是模型生成的问题还是下游二次解析造成的破坏。QuoteBench用56个任务把两者拆开:把同一条模型生成的回复分别在原始路径和额外加了一个故意不做转义处理的解析器的路径上重放,结果发现仅加这一层解析,成功率就在55.4到73.2个百分点之间下降;而事先告知模型这个边界的存在,能在八组配置中的六组里挽回30.4到60.7个百分点。以GPT-5.6-sol为例,常规匹配分数几乎没变(-3.6分),但背后其实是-64.3分的破坏和+60.7分的补偿相互抵消的结果。

他们做了什么

问题所在:智能体生成的命令常常要经过ssh远程执行、容器执行(docker exec sh -c)、或CI流水线等环节被重新包进一层双引号里解析,普通的匹配成功率无法区分这是模型生成错了,还是命令在下游被这层重新解析破坏了。
方法:作者构建了QuoteBench,包含来自14个真实事故案例总结出的14类操作、共56个一次性Bash任务,把模型生成命令的约定方式(生成契约)和命令实际执行的传输路径(执行传输)做交叉实验。他们把同一条已生成的回复分别放到原始路径和故意加了一个未转义解析器的嵌套路径上重放,并测试了一种提前告知模型存在该边界的契约。
结果:在最前沿模型上,原始路径执行的成功率已接近饱和(91.1%到100%),但只要多加这一层重新解析,成功率就会在八组配置中普遍下降55.4到73.2个百分点;而提前告知模型这个边界,能在其中六组配置里挽回30.4到60.7个百分点,另外两组则没有改善甚至略微变差。
关键发现:模型的表面匹配分数可能具有误导性。GPT-5.6-sol的匹配分数变化仅为-3.6分,看似几乎没受影响,但实际上是-64.3分的严重破坏和+60.7分的大幅补偿正好相互抵消。仅仅改变执行路径,就足以让26对可比较模型中至少一对的排名发生明确反转。
实践启示:正确的转义处理或改用临时脚本都能完全消除这个问题,但作者对六个真实公开智能体系统的调查发现,这层引号解析边界往往是在模型输出契约之外的下游环节处理的,单靠模型自身并不总能解决。

Figure 1: Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract-then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The RR and NN cells are matched. The RN cell measures fixed-reply damage, and NN−RN is the realized contract-conditioned contrast. Panel (b) works the decomposition for GPT-5.6-sol.

Table 1: One worked QuoteBench task (write-file, tier 3). The ⟨⟨⋅⟩⟩ markers delimit the exact literal payload and are not part of it. The ↩ symbol marks a newline that the payload itself contains. The validator compares exact final bytes, so any command that reaches the target state passes. Appendix Table 7 maps the full family set to the covered mechanisms. The artifact contains all 56 instructions, fixtures, oracles, and validators.
Instruction	Create a file named out.txt whose content is exactly the text ⟨⟨back\slash and `date` here↩ ’single’ "double" $(pwd) 100%⟩⟩ followed by a single trailing newline.
Hazards	backslash, backtick, command substitution, multiline, mixed quotes
Naive command	echo "back\slash and `date` here …" > out.txt; the backticks execute date, $(pwd) expands locally, and the payload’s inner double quote terminates the string early
Final-state gate	exact bytes of out.txt, including one trailing newline, with no collateral files
One passing command	a single-quoted printf with an embedded-quote splice (the machine oracle). Any other correct encoding receives equal credit

Figure 2: Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix Table 13 lists the queried effort settings and request parameters. Table 2 separately summarizes the best observed setting per base model. Columns follow the mechanism groups in Appendix Table 7. SSH-like tasks use the local two-shell simulation. Rows are ordered by matched-nested score in this frozen sweep, which differs from the best-observed order in Table 2.

Table 2: Best-observed QuoteBench scorecard. Each base model contributes the measured setting with the highest matched-nested All-56 score. Ties prefer default, then lower mean provider-reported output tokens. default means that the request omitted the effort field. The number and names of measured settings differ by provider and appear in Figure 4 and Appendix B. This table is descriptive: each cell is one stored trial-0 generation and the selection is a within-model maximum over rungs; Appendix E.1.6 reports draw-to-draw spread. The Qwen rows expose only a think toggle rather than an effort ladder, so their best-observed setting is taken from the same-window sweep; for Qwen3.5-27B, the only Qwen size with a ladder row in Table 12, the resulting one-task difference is serving-window drift. Fixed same-window configurations support the mechanism analysis in Table 4.
Model	Best observed setting	Control	Hostile	All 56 (%)	Hostile LOFO range (%)
GPT-5.5	xhigh	14/14	42/42	100.0	[100.0, 100.0]
Opus-5	xhigh	14/14	42/42	100.0	[100.0, 100.0]
Fable-5	max	14/14	42/42	100.0	[100.0, 100.0]
GPT-5.6-sol	high	14/14	41/42	98.2	[97.4, 100.0]
Opus-4.8	max	12/14	41/42	94.6	[97.4, 100.0]
Gemini-3.1-Pro	low	14/14	37/42	91.1	[87.2, 89.7]
Sonnet-4.6	high	9/14	27/42	64.3	[61.5, 69.2]
Gemini-3.5-Flash	medium	10/14	26/42	64.3	[59.0, 66.7]
Haiku-4.5	medium	8/14	13/42	37.5	[28.2, 33.3]
Qwen3.5-27B	non-think	8/14	9/42	30.4	[15.4, 23.1]
Qwen3.5-4B	think	5/14	7/42	21.4	[10.3, 17.9]
Qwen3.5-9B	non-think	5/14	5/42	17.9	[5.1, 12.8]
Gemini-3.1-Flash-Lite	default	6/14	2/42	14.3	[2.6, 5.1]

Figure 3: Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The RR value is raw success, RN is the nested-replay pass rate of raw-conditioned replies, and NN is matched nested success. The NN−RN gap is compensation. Appendix Table 14 reports every rung, including the byte-identical Gemini-3.1-Flash-Lite settings.

Table 3: Absolute GNU replay pass rates in the separate Study-B campaign, pooled over each model’s measured effort ladder and three trials per cell.
Model	Attempts	Raw (%)	Native (%)	Δ (pp)
Opus-4.8	840	95.4	98.0	+2.6
Opus-5	840	98.2	97.4	−0.8
Fable-5	840	99.3	97.1	−2.1
Gemini-3.1-Pro	504	98.8	95.0	−3.8
GPT-5.6-sol	672	96.9	94.3	−2.5
Gemini-3.5-Flash	672	95.7	85.7	−10.0

Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched-nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order. Backward or dominated segments show measured non-monotonicity. Panels group model families for legibility, and Qwen sizes contribute two-point think-toggle trajectories.

Table 4: All four crossover cells for the eight same-window configurations. Cell notation follows Figure 1: generation contract precedes transport. Cells are pass rates (%). Effects are percentage points. Damage is RN−RR, compensation is NN−RN, and the matched gap is NN−RR. Appendix Table 14 reports the interaction at every measured rung, and Table 15 gives the sensitivity tests.
Model	RR	RN	NR	NN	Damage	Comp.	Matched gap
GPT-5.6-sol	94.6	30.4	55.4	91.1	−64.3	+60.7	−3.6
GPT-5.5	100.0	28.6	50.0	89.3	−71.4	+60.7	−10.7
Opus-5	96.4	30.4	42.9	89.3	−66.1	+58.9	−7.1
Gemini-3.1-Pro	98.2	25.0	33.9	80.4	−73.2	+55.4	−17.9
Gemini-3.5-Flash	96.4	28.6	67.9	58.9	−67.9	+30.4	−37.5
Opus-4.8	91.1	26.8	62.5	57.1	−64.3	+30.4	−33.9
Qwen3.5-27B	85.7	30.4	83.9	30.4	−55.4	0.0	−55.4
Gemini-3.1-Flash-Lite	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3

Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered scale. Blank cells mark rungs the provider does not expose. Models follow descending GNU replay campaign effect. Labels are not comparable compute budgets across providers.

Table 5: Private-v2 crossover on 42 hostile payloads under the single-clause disclosed-boundary contract. The private set is hostile-only and not difficulty-matched to the public core, so absolute rates are interpreted within this set. Cells are pass rates. Effects are percentage points. Damage is RN−RR, compensation is NN−RN, interaction is (NN−NR)−(RN−RR), and the matched gap is NN−RR.
Model	RR	RN	NR	NN	Damage	Comp.	Interaction	Matched gap
GPT-5.6-sol	92.9	19.0	50.0	97.6	−73.8	+78.6	+121.4	+4.8
Opus-4.8	92.9	16.7	59.5	42.9	−76.2	+26.2	+59.5	−50.0

Table 6: Private-v1 fixed raw replies under three transports on 42 tasks. This campaign predates the private-v2 crossover in Table 5. The two tables therefore use different generations. The first three columns report tasks passed out of 42. Script gain is the temporary-script rate minus the nested-wrapper rate, in percentage points.
Model	Raw	Nested wrapper	Temporary script	Script gain (pp)
GPT-5.6-sol	41/42	8/42	41/42	+78.6
Opus-4.8	40/42	7/42	40/42	+78.6
Qwen3.5-27B	30/42	9/42	30/42	+50.0

Table 7: Mechanism groups used to construct the frozen core; the mapping documents coverage of the surveyed failure mechanisms. Benchmark analyses weight the 14 operation families equally.
Mechanism group	Representative failures	QuoteBench families
Literal quote and expansion	Apostrophes, double quotes, dollars, backticks, multiline payloads	write-file, JSON writing, Git commit, environment passing, heredoc writing
Word splitting and path semantics	Spaces, globs, leading dashes, hostile filenames, argument boundaries	argv passing, hostile filenames, find/glob, bulk rename
Embedded-language escaping	Regex versus literal matching, sed replacement, AWK string processing	grep count, sed replace, field lookup, JSON writing
Second parser or remote-like expansion	Local expansion before a second shell, argument joining, heredoc transport	SSH-like nested execution, SSH-like heredoc
Command-boundary representation	Command string, shell stdin, temporary file, argv, provider tool schema	raw/nested crossover, native-tool study, script bypass, typed pilot

Table 8: Command boundaries in six public agent systems, inspected at fixed commits. Contract is what the system asks the model to produce. A nested boundary arises on a separate axis from what the command targets. Observed boundary is what the harness then does with the reply. The classification concerns the parser boundary only. Sources: 20; 27; 13; 28; 22; 18.
System	Contract	Source anchor	Observed boundary
Codex	native	core/src/shell.rs:20--30, commit fa1d4c4	command string becomes shell -c/-lc R
SWE-agent	raw	agents.py:936--967 and swe_env.py:197--222, commit 3ea751c	agent action enters a persistent Bash session
LangChain	native	shell_tool.py:217--232,491--515, commit b3a6d9a	structured command: string is written to shell stdin
Terminal-Bench	raw	tmux_session.py:26--33,75--173, commit d28711d	command/key strings enter an interactive shell through tmux
OpenHands	native	acp-command.ts:11--45,65--147, commit 850bd64	human-readable command is tokenized to argv, and spawn has no shell
AutoGen	native	_docker_code_executor.py:327--363, commit 027ecf0	generated code is written to a temporary file and invoked by argv

Table 9: Real-ssh grounding. Each stored raw reply is replayed through bash -c and through a real ssh localhost "R" remote command; ssh damage is the second minus the first. It matches the synthetic nested damage to the decimal for seven of eight configurations; the disclosed-reply cells are grounded in Appendix Table 21. Zero model calls.
Configuration	Raw	ssh	ssh damage	Nested damage
GPT-5.6-sol	94.6	30.4	−64.3	−64.3
GPT-5.5	100.0	28.6	−71.4	−71.4
Opus-5	96.4	30.4	−66.1	−66.1
Gemini-3.1-Pro	98.2	26.8	−71.4	−73.2
Gemini-3.5-Flash	96.4	28.6	−67.9	−67.9
Opus-4.8	91.1	26.8	−64.3	−64.3
Qwen3.5-27B	85.7	30.4	−55.4	−55.4
Gemini-3.1-Flash-Lite	78.6	19.6	−58.9	−58.9

Table 10: Campaign map. All replays are zero-call executions of stored replies in the pinned container.
Campaign	Generations	Design	Feeds
Study A same-window sweep	8 configs × 56 × 2 contracts	one randomized window, effort unset	Tables 4, 15
Study A ladder sweep	44 rungs, 11 configs	per-provider windows	Tables 2, 12
Study A rung crossover	30 rungs, 7 configs, 26 crossover pts	replay both transports	Table 14, Fig. 3
Public three-draw repeat	8 configs × 56 × 2 × 2 draws	same design as the sweep	Appendix E.1.6
Study B native tool	8,736 generations, 17,472 replays	observational, both userlands	Tables 3, 17, 18
Private-v2 crossover	2 models × 42 hostile payloads	one serving window	Table 5
Private-v1 replay	3 models × 42 tasks	earlier generations; script bypass	Table 6
BSD-live legacy	6 configurations	BSD-elicited, GNU-replayed	Table 16
Real-ssh grounding	8 configs × 56	ssh localhost replay	Table 9
Advice arm	8 configs × 56 × 2	same-window paired advice contrast	Table 19
Grammar crossover	8 configs × 56 × 2 disclosed	replay-only wrapper 2×2	Table 20
Real-ssh full crossover	6 configs × 56 × 2	disclosed replies on real ssh	Table 21
JSON boundary	6 configs × 56	serializer replay	Table 22

Table 11: Calibration of the unset-effort sweep arm against each configuration’s labelled ladder. Distance is unset success minus the lowest-rung success, in percentage points. The two measurements come from different serving windows, so small differences are descriptive only.
Model	Unset (%)	Nearest rung	Distance to lowest rung
GPT-5.6-sol	91.1	low	0.0
GPT-5.5	89.3	high	−5.4
Opus-5	89.3	medium	+28.6
Gemini-3.1-Pro	80.4	none within the ladder	−10.7
Gemini-3.5-Flash	58.9	low	+1.8
Opus-4.8	57.1	xhigh	+17.9
Gemini-3.1-Flash-Lite	14.3	all four rungs tie	0.0

Table 12: Matched-nested effort ladders under the disclosed-boundary contract. Within each row, success rates and mean provider-reported output tokens follow the setting order in the second column. Each point uses trial 0 over 56 tasks.
Model	Settings (in order)	Success (%)	Mean output tokens
GPT-5.6-sol	low / medium / high / xhigh	91.1 / 92.9 / 98.2 / 98.2	362 / 565 / 773 / 1,073
GPT-5.5	low / medium / high / xhigh	94.6 / 92.9 / 91.1 / 100.0	507 / 655 / 1,164 / 2,757
Opus-5	low / medium / high / xhigh / max	60.7 / 91.1 / 96.4 / 100.0 / 98.2	796 / 1,336 / 1,458 / 2,421 / 3,499
Fable-5	low / medium / high / xhigh / max	71.4 / 75.0 / 96.4 / 92.9 / 100.0	332 / 569 / 843 / 1,212 / 2,396
Opus-4.8	low / medium / high / xhigh / max	39.3 / 48.2 / 50.0 / 62.5 / 94.6	835 / 1,291 / 1,706 / 3,481 / 6,960
Gemini-3.1-Pro	low / medium / high	91.1 / 91.1 / 91.1	5,267 / 5,753 / 6,308
Sonnet-4.6	low / medium / high / xhigh	23.2 / 28.6 / 64.3 / 62.5	28 / 1,411 / 4,337 / 4,539
Gemini-3.5-Flash	minimal / low / medium / high	57.1 / 58.9 / 64.3 / 62.5	3,851 / 3,108 / 3,081 / 3,389
Haiku-4.5	low / medium / high / xhigh	32.1 / 37.5 / 32.1 / 26.8	5,432 / 4,717 / 4,708 / 4,783
Qwen3.5-27B	non-thinking / thinking	28.6 / 28.6	20 / 7,489
Gemini-3.1-Flash-Lite	minimal / low / medium / high	14.3 / 14.3 / 14.3 / 14.3	20 / 20 / 20 / 20

Table 13: Study-A model identifiers and request parameters. The effort column lists exactly the settings queried; sweep arms omitted the effort field. Temperature is reported only where the interface accepts it.
Display name	Model identifier	Effort settings queried	Decoding parameters
GPT-5.6-sol	gpt-5.6-sol	low, medium, high, xhigh	max output tokens 16,000, temperature not sent
GPT-5.5	gpt-5.5	low, medium, high, xhigh	max output tokens 16,000, temperature not sent
Opus-5	claude-opus-5	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Opus-4.8	claude-opus-4-8	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Fable-5	claude-fable-5	low, medium, high, xhigh, max	provider defaults, no sampling or length control sent
Sonnet-4.6	claude-sonnet-4-6	low, medium, high, xhigh	provider defaults, no sampling or length control sent
Haiku-4.5	claude-haiku-4-5	low, medium, high, xhigh	provider defaults, no sampling or length control sent
Gemini-3.1-Pro	gemini-3.1-pro-preview	low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.5-Flash	gemini-3.5-flash	minimal, low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.1-Flash-Lite	gemini-3.1-flash-lite-preview	minimal, low, medium, high	temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Qwen3.5-27B	Qwen/Qwen3.5-27B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-9B	Qwen/Qwen3.5-9B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-4B	Qwen/Qwen3.5-4B	non-thinking, thinking	temperature 0, max tokens 4,096 non-thinking, omitted thinking

Table 14: Crossover at every measured rung. Cells are percentages and effects are percentage points, defined as in Table 4. An asterisk marks the descriptive masked-fragility rule. Gemini-3.1-Flash-Lite returns byte-identical replies at all four settings and is retained only to document that the provider did not expose a usable ladder.
Configuration	Rung	RR	RN	NR	NN	Damage	Compensation	Matched gap
GPT-5.6-sol	low	98.2	28.6	53.6	91.1	−69.6	+62.5	−7.1
GPT-5.6-sol	medium	94.6	30.4	48.2	92.9	−64.3	+62.5	−1.8∗
GPT-5.6-sol	high	98.2	32.1	48.2	98.2	−66.1	+66.1	+0.0∗
GPT-5.6-sol	xhigh	98.2	30.4	51.8	98.2	−67.9	+67.9	+0.0∗
Opus-5	low	98.2	28.6	57.1	60.7	−69.6	+32.1	−37.5
Opus-5	medium	100.0	32.1	41.1	91.1	−67.9	+58.9	−8.9
Opus-5	high	96.4	30.4	42.9	96.4	−66.1	+66.1	+0.0∗
Opus-5	xhigh	98.2	32.1	42.9	100.0	−66.1	+67.9	+1.8∗
Opus-5	max	100.0	32.1	46.4	98.2	−67.9	+66.1	−1.8∗
Gemini-3.1-Pro	low	100.0	25.0	32.1	91.1	−75.0	+66.1	−8.9
Gemini-3.1-Pro	medium	98.2	26.8	41.1	91.1	−71.4	+64.3	−7.1
Gemini-3.1-Pro	high	98.2	26.8	35.7	91.1	−71.4	+64.3	−7.1
Fable-5	low	100.0	30.4	48.2	71.4	−69.6	+41.1	−28.6
Fable-5	medium	100.0	30.4	48.2	75.0	−69.6	+44.6	−25.0
Fable-5	high	98.2	32.1	42.9	96.4	−66.1	+64.3	−1.8∗
Fable-5	xhigh	100.0	33.9	46.4	92.9	−66.1	+58.9	−7.1
Fable-5	max	100.0	30.4	35.7	100.0	−69.6	+69.6	+0.0∗
Gemini-3.5-Flash	minimal	96.4	23.2	75.0	57.1	−73.2	+33.9	−39.3
Gemini-3.5-Flash	low	92.9	25.0	67.9	58.9	−67.9	+33.9	−33.9
Gemini-3.5-Flash	medium	96.4	23.2	76.8	64.3	−73.2	+41.1	−32.1
Gemini-3.5-Flash	high	96.4	23.2	71.4	62.5	−73.2	+39.3	−33.9
Opus-4.8	low	87.5	28.6	69.6	39.3	−58.9	+10.7	−48.2
Opus-4.8	medium	92.9	28.6	67.9	48.2	−64.3	+19.6	−44.6
Opus-4.8	high	96.4	33.9	64.3	50.0	−62.5	+16.1	−46.4
Opus-4.8	xhigh	98.2	32.1	60.7	62.5	−66.1	+30.4	−35.7
Opus-4.8	max	98.2	30.4	44.6	94.6	−67.9	+64.3	−3.6∗
Gemini-3.1-Flash-Lite	minimal	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	low	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	medium	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3
Gemini-3.1-Flash-Lite	high	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3

Table 15: Enumerated and Holm-adjusted two-sided family-sign p values for the two primary Study-A components under the single-clause disclosed-boundary contract. Effect sizes are percentage points.
	Fixed-reply transport	Contract-conditioned compensation
Model	Effect [95% CI]	Enum. p	Holm p	Effect [95% CI]	Enum. p	Holm p
GPT-5.6-sol	−64.3 [−80.4,−46.4]	.000244	.001465	+60.7 [+46.4,+75.0]	.000244	.001953
GPT-5.5	−71.4 [−85.7,−55.4]	.000244	.001465	+60.7 [+44.6,+75.0]	.000244	.001953
Opus-5	−66.1 [−82.1,−50.0]	.000244	.001465	+58.9 [+41.1,+75.0]	.000488	.002930
Gemini-3.1-Pro	−73.2 [−87.5,−58.9]	.000122	.000977	+55.4 [+33.9,+73.2]	.001221	.006104
Gemini-3.5-Flash	−67.9 [−82.1,−51.8]	.000244	.001465	+30.4 [+12.5,+48.2]	.013672	.041016
Opus-4.8	−64.3 [−80.4,−46.4]	.000244	.001465	+30.4 [+14.3,+48.2]	.003906	.015625
Qwen3.5-27B	−55.4 [−69.6,−41.1]	.000244	.001465	0.0 [0.0,0.0]	1.000000	1.000000
Gemini-3.1-Flash-Lite	−58.9 [−71.4,−48.2]	.000122	.000977	−5.4 [−10.7,0.0]	.250000	.500000

Table 16: Cross-userland transfer in the earlier BSD-live campaign. Each cell reports pass rate on BSD live execution and GNU replay of the identical stored command. Arms retain their original campaign multiplicity: raw cells contain 56 records, while nested cells contain 56 or 168 depending on the configuration. We interpret only the within-arm BSD→GNU change. The final column summarizes the higher-transfer userland; it is not a counterfactual GNU-targeted generation result.
Model	Raw BSD→GNU	Nested BSD→GNU	Transfers better to
Fable-5	→91.1	→87.5	BSD
Qwen3.5-27B (non-thinking)	→87.5	→32.1	GNU
Qwen3.5-27B (thinking)	→82.1	→50.0	GNU
Gemini-3.1-Pro	→100.0	→96.4	GNU
Gemini-3.5-Flash	→96.4	→67.9	BSD
Opus-4.8	→87.5	→73.8	BSD (raw); tie (nested)

Table 17: Exploratory native-minus-raw effects over each measured effort ladder, ordered by GNU replay effect.
	BSD live	GNU replay
Model	Δ	LOFO	PF/FP	Δ	LOFO	PF/FP
Opus-4.8	+3.10	[+2.18,+3.46]	5/31	+2.62	[+1.67,+2.95]	11/33
Opus-5	−0.36	[−0.51,0.00]	12/9	−0.83	[−1.03,−0.51]	16/9
Fable-5	+0.95	[0.00,+1.15]	3/11	−2.14	[−2.44,−0.64]	22/4
GPT-5.6-sol	−1.19	[−1.92,−0.48]	25/17	−2.53	[−3.37,−0.64]	26/9
Gemini-3.1-Pro	+1.98	[−0.64,+3.21]	8/18	−3.77	[−4.91,−0.64]	23/4
Gemini-3.5-Flash	−4.02	[−4.33,−3.04]	37/10	−9.97	[−10.74,−6.89]	77/10

Table 18: Study-B execution outcomes, aggregated across six models. Each row partitions 4,368 executions. Adherence denotes an invalid one-call tool invocation. Syntax includes parser and command-usage errors. Exit-0 wrong is a silent final-state failure.
Userland	Contract	Pass	Adherence	Syntax	Nonzero	Exit-0 wrong
BSD	raw	4230	0	31	59	48
BSD	native	4236	20	9	41	62
GNU	raw	4252	0	39	36	41
GNU	native	4146	20	22	128	52

Table 19: Matched nested success (of 56) under the disclosed contract without and with escaping advice, paired within one serving window. Advice barely moves the top of the ladder, moves the middle of the ladder the most, and does not reach the bottom.
Configuration	Advice-free	With advice	Δ (pp)
GPT-5.6-sol	54	49	−8.9
GPT-5.5	49	53	+7.1
Opus-5	52	54	+3.6
Sonnet-4.6	32	46	+25.0
Haiku-4.5	25	32	+12.5
Opus-4.8	30	34	+7.1
Qwen3.5-27B	17	18	+1.8
Gemini-3.1-Flash-Lite	7	8	+1.8

Table 20: Grammar crossover (tasks passed of 56). Rows are the disclosed grammar, columns the executed wrapper; the diagonal is matched, the anti-diagonal mismatched. The diagonal advantage separates the same top, middle, and bottom groups as the matched-nested scores.
	Double disclosed	Single disclosed	Diag−anti
Configuration	on double	on single	on single	on double	(pp)
GPT-5.6-sol	54	7	53	10	+80.4
GPT-5.5	49	8	54	8	+77.7
Opus-5	52	8	45	16	+65.2
Sonnet-4.6	32	11	23	16	+25.0
Opus-4.8	30	11	23	18	+21.4
Haiku-4.5	25	13	20	11	+18.8
Qwen3.5-27B	17	11	11	17	+0.0
Gemini-3.1-Flash-Lite	7	19	9	19	−19.6

Table 21: Real-ssh full crossover (tasks passed of 56). Damage is RN−RR and compensation is NN−RN, both on the real ssh path. The final column repeats the synthetic nested compensation for comparison.
Configuration	RR	RN	NR	NN	ssh comp.	Nested comp.
GPT-5.6-sol	53	17	31	51	+60.7	+60.7
GPT-5.5	56	16	28	50	+60.7	+60.7
Opus-5	54	17	24	50	+58.9	+58.9
Opus-4.8	51	15	35	32	+30.4	+30.4
Qwen3.5-27B	48	17	47	17	+0.0	+0.0
Gemini-3.1-Flash-Lite	44	11	45	10	−1.8	−5.4

Table 22: JSON serializer boundary (tasks passed of 56). Correct-serializer damage is ≈0; naive-embedding damage is comparable to the shell nested transport. Unparseable counts replies whose naive JSON embedding fails to parse.
Configuration	Raw	Correct	Naive	Naive damage
GPT-5.6-sol	53	53	21	−57.1
GPT-5.5	56	56	19	−66.1
Opus-5	54	54	22	−57.1
Opus-4.8	51	51	18	−58.9
Qwen3.5-27B	48	48	19	−51.8
Gemini-3.1-Flash-Lite	44	44	8	−64.3

为什么重要

在实际部署中,智能体经常通过ssh、容器或CI流水线发送命令,这正是本文所研究的二次解析边界会出现的地方,因此忽略这一点的排行榜可能无法反映模型部署后的真实表现。任何评估或上线会下达命令的编程智能体的人,都应该关注一个分数背后所依赖的执行路径和生成契约,而不只是分数本身。

本文术语

匹配分数(matched score) · 直接执行模型生成命令后得到的常规成功率,没有拆分失败究竟来自哪个环节
生成契约(generation contract) · 告诉模型应该以什么格式或前提生成命令的约定
执行传输(execution transport) · 生成的命令在真正到达shell执行之前所经过的路径,例如直接执行或被重新包裹后执行
嵌套传输(nested transport) · 人为加入的一种测试条件,给命令再套一层双引号解析,模拟ssh、容器或CI包装器对命令的二次解析
最终状态验证(final-state validation) · 只根据执行后文件、参数或仓库状态是否与预期完全一致来打分,不关心命令是如何达到该状态的

论文原文摘要（英文）

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

作者 · Shangao Li

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Shangao Li et al., arXiv:2608.13547, CC BY 4.0

QuoteBench: How Matched Scores Can Hide Command-Path Failures

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道