QuoteBench: How Matched Scores Can Hide Command-Path Failures
arXiv:2608.135472026-08-12
AI coding agents' benchmark scores can hide the fact that a working command breaks the moment it's re-parsed downstream
Bash commands generated by LLM coding agents are often serialized and re-parsed by intermediate layers (like ssh, container, or CI wrappers) before they actually run, and ordinary matched-score benchmarks can't tell whether a failure came from bad generation or from that downstream re-parsing. QuoteBench uses 56 tasks to separate the two by replaying the exact same generated reply through both a raw path and a path with one deliberately unescaped parser added, finding that this single added parser drops success by 55.4 to 73.2 percentage points, while simply telling the model about the boundary recovers 30.4 to 60.7 points in six of eight configurations. For GPT-5.6-sol, the ordinary matched score barely moves (-3.6 points), masking a -64.3-point loss from the boundary and a +60.7-point recovery from disclosure that cancel each other out.
What they did
The problem: commands an LLM agent writes often get wrapped and re-interpreted by another layer before execution, such as ssh host "…", docker exec sh -c "…", or a CI step, and plain pass/fail scores can't reveal whether that re-parsing, not the model, caused the failure.
Method: the authors built QuoteBench, 56 one-shot Bash tasks across 14 operation families drawn from real incident reports, and crossed the way a model is told to produce a command (generation contract) with the path that command travels before running (execution transport). They replayed the exact same stored reply once raw and once through one deliberately unescaped added parser, and also tested a 'disclosed-boundary' contract that tells the model in advance its reply will be dropped inside bash -c "…".
Result: at the frontier, raw execution is already nearly saturated (91.1-100% success), but adding just one re-parsing step drops success by 55.4 to 73.2 points across eight tested configurations. Warning the model about that boundary in advance recovers 30.4 to 60.7 points in six configurations, while two configurations show no recovery or a slight drop.
Key finding: a model's headline matched score can be misleading. GPT-5.6-sol shows almost no change (-3.6 points) under standard scoring, but that number hides a large -64.3-point loss from the boundary and a large +60.7-point recovery from disclosure that happen to cancel out. Changing the execution path alone was enough to flip the ranking of at least one model pair among 26 compared.
Practical angle: correct escaping or using a temporary script both eliminate the problem entirely, but a survey of six real public agent systems found this quoting boundary is often handled downstream of the model's stated output contract, meaning the model alone can't always fix it.
Figure 1: Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract-then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The RR and NN cells are matched. The RN cell measures fixed-reply damage, and NN−RN is the realized contract-conditioned contrast. Panel (b) works the decomposition for GPT-5.6-sol.
Table 1: One worked QuoteBench task (write-file, tier 3). The ⟨⟨⋅⟩⟩ markers delimit the exact literal payload and are not part of it. The ↩ symbol marks a newline that the payload itself contains. The validator compares exact final bytes, so any command that reaches the target state passes. Appendix Table 7 maps the full family set to the covered mechanisms. The artifact contains all 56 instructions, fixtures, oracles, and validators.
Instruction
Create a file named out.txt whose content is exactly the text ⟨⟨back\slash and `date` here↩ ’single’ "double" $(pwd) 100%⟩⟩ followed by a single trailing newline.
echo "back\slash and `date` here …" > out.txt; the backticks execute date, $(pwd) expands locally, and the payload’s inner double quote terminates the string early
Final-state gate
exact bytes of out.txt, including one trailing newline, with no collateral files
One passing command
a single-quoted printf with an embedded-quote splice (the machine oracle). Any other correct encoding receives equal credit
Figure 2: Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix Table 13 lists the queried effort settings and request parameters. Table 2 separately summarizes the best observed setting per base model. Columns follow the mechanism groups in Appendix Table 7. SSH-like tasks use the local two-shell simulation. Rows are ordered by matched-nested score in this frozen sweep, which differs from the best-observed order in Table 2.
Table 2: Best-observed QuoteBench scorecard. Each base model contributes the measured setting with the highest matched-nested All-56 score. Ties prefer default, then lower mean provider-reported output tokens. default means that the request omitted the effort field. The number and names of measured settings differ by provider and appear in Figure 4 and Appendix B. This table is descriptive: each cell is one stored trial-0 generation and the selection is a within-model maximum over rungs; Appendix E.1.6 reports draw-to-draw spread. The Qwen rows expose only a think toggle rather than an effort ladder, so their best-observed setting is taken from the same-window sweep; for Qwen3.5-27B, the only Qwen size with a ladder row in Table 12, the resulting one-task difference is serving-window drift. Fixed same-window configurations support the mechanism analysis in Table 4.
Model
Best observed setting
Control
Hostile
All 56 (%)
Hostile LOFO range (%)
GPT-5.5
xhigh
14/14
42/42
100.0
[100.0, 100.0]
Opus-5
xhigh
14/14
42/42
100.0
[100.0, 100.0]
Fable-5
max
14/14
42/42
100.0
[100.0, 100.0]
GPT-5.6-sol
high
14/14
41/42
98.2
[97.4, 100.0]
Opus-4.8
max
12/14
41/42
94.6
[97.4, 100.0]
Gemini-3.1-Pro
low
14/14
37/42
91.1
[87.2, 89.7]
Sonnet-4.6
high
9/14
27/42
64.3
[61.5, 69.2]
Gemini-3.5-Flash
medium
10/14
26/42
64.3
[59.0, 66.7]
Haiku-4.5
medium
8/14
13/42
37.5
[28.2, 33.3]
Qwen3.5-27B
non-think
8/14
9/42
30.4
[15.4, 23.1]
Qwen3.5-4B
think
5/14
7/42
21.4
[10.3, 17.9]
Qwen3.5-9B
non-think
5/14
5/42
17.9
[5.1, 12.8]
Gemini-3.1-Flash-Lite
default
6/14
2/42
14.3
[2.6, 5.1]
Figure 3: Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The RR value is raw success, RN is the nested-replay pass rate of raw-conditioned replies, and NN is matched nested success. The NN−RN gap is compensation. Appendix Table 14 reports every rung, including the byte-identical Gemini-3.1-Flash-Lite settings.
Table 3: Absolute GNU replay pass rates in the separate Study-B campaign, pooled over each model’s measured effort ladder and three trials per cell.
Model
Attempts
Raw (%)
Native (%)
Δ (pp)
Opus-4.8
840
95.4
98.0
+2.6
Opus-5
840
98.2
97.4
−0.8
Fable-5
840
99.3
97.1
−2.1
Gemini-3.1-Pro
504
98.8
95.0
−3.8
GPT-5.6-sol
672
96.9
94.3
−2.5
Gemini-3.5-Flash
672
95.7
85.7
−10.0
Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched-nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order. Backward or dominated segments show measured non-monotonicity. Panels group model families for legibility, and Qwen sizes contribute two-point think-toggle trajectories.
Table 4: All four crossover cells for the eight same-window configurations. Cell notation follows Figure 1: generation contract precedes transport. Cells are pass rates (%). Effects are percentage points. Damage is RN−RR, compensation is NN−RN, and the matched gap is NN−RR. Appendix Table 14 reports the interaction at every measured rung, and Table 15 gives the sensitivity tests.
Model
RR
RN
NR
NN
Damage
Comp.
Matched gap
GPT-5.6-sol
94.6
30.4
55.4
91.1
−64.3
+60.7
−3.6
GPT-5.5
100.0
28.6
50.0
89.3
−71.4
+60.7
−10.7
Opus-5
96.4
30.4
42.9
89.3
−66.1
+58.9
−7.1
Gemini-3.1-Pro
98.2
25.0
33.9
80.4
−73.2
+55.4
−17.9
Gemini-3.5-Flash
96.4
28.6
67.9
58.9
−67.9
+30.4
−37.5
Opus-4.8
91.1
26.8
62.5
57.1
−64.3
+30.4
−33.9
Qwen3.5-27B
85.7
30.4
83.9
30.4
−55.4
0.0
−55.4
Gemini-3.1-Flash-Lite
78.6
19.6
80.4
14.3
−58.9
−5.4
−64.3
Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered scale. Blank cells mark rungs the provider does not expose. Models follow descending GNU replay campaign effect. Labels are not comparable compute budgets across providers.
Table 5: Private-v2 crossover on 42 hostile payloads under the single-clause disclosed-boundary contract. The private set is hostile-only and not difficulty-matched to the public core, so absolute rates are interpreted within this set. Cells are pass rates. Effects are percentage points. Damage is RN−RR, compensation is NN−RN, interaction is (NN−NR)−(RN−RR), and the matched gap is NN−RR.
Model
RR
RN
NR
NN
Damage
Comp.
Interaction
Matched gap
GPT-5.6-sol
92.9
19.0
50.0
97.6
−73.8
+78.6
+121.4
+4.8
Opus-4.8
92.9
16.7
59.5
42.9
−76.2
+26.2
+59.5
−50.0
Table 6: Private-v1 fixed raw replies under three transports on 42 tasks. This campaign predates the private-v2 crossover in Table 5. The two tables therefore use different generations. The first three columns report tasks passed out of 42. Script gain is the temporary-script rate minus the nested-wrapper rate, in percentage points.
Model
Raw
Nested wrapper
Temporary script
Script gain (pp)
GPT-5.6-sol
41/42
8/42
41/42
+78.6
Opus-4.8
40/42
7/42
40/42
+78.6
Qwen3.5-27B
30/42
9/42
30/42
+50.0
Table 7: Mechanism groups used to construct the frozen core; the mapping documents coverage of the surveyed failure mechanisms. Benchmark analyses weight the 14 operation families equally.
raw/nested crossover, native-tool study, script bypass, typed pilot
Table 8: Command boundaries in six public agent systems, inspected at fixed commits. Contract is what the system asks the model to produce. A nested boundary arises on a separate axis from what the command targets. Observed boundary is what the harness then does with the reply. The classification concerns the parser boundary only. Sources: 20; 27; 13; 28; 22; 18.
System
Contract
Source anchor
Observed boundary
Codex
native
core/src/shell.rs:20--30, commit fa1d4c4
command string becomes shell -c/-lc R
SWE-agent
raw
agents.py:936--967 and swe_env.py:197--222, commit 3ea751c
agent action enters a persistent Bash session
LangChain
native
shell_tool.py:217--232,491--515, commit b3a6d9a
structured command: string is written to shell stdin
Terminal-Bench
raw
tmux_session.py:26--33,75--173, commit d28711d
command/key strings enter an interactive shell through tmux
OpenHands
native
acp-command.ts:11--45,65--147, commit 850bd64
human-readable command is tokenized to argv, and spawn has no shell
AutoGen
native
_docker_code_executor.py:327--363, commit 027ecf0
generated code is written to a temporary file and invoked by argv
Table 9: Real-ssh grounding. Each stored raw reply is replayed through bash -c and through a real ssh localhost "R" remote command; ssh damage is the second minus the first. It matches the synthetic nested damage to the decimal for seven of eight configurations; the disclosed-reply cells are grounded in Appendix Table 21. Zero model calls.
Configuration
Raw
ssh
ssh damage
Nested damage
GPT-5.6-sol
94.6
30.4
−64.3
−64.3
GPT-5.5
100.0
28.6
−71.4
−71.4
Opus-5
96.4
30.4
−66.1
−66.1
Gemini-3.1-Pro
98.2
26.8
−71.4
−73.2
Gemini-3.5-Flash
96.4
28.6
−67.9
−67.9
Opus-4.8
91.1
26.8
−64.3
−64.3
Qwen3.5-27B
85.7
30.4
−55.4
−55.4
Gemini-3.1-Flash-Lite
78.6
19.6
−58.9
−58.9
Table 10: Campaign map. All replays are zero-call executions of stored replies in the pinned container.
Campaign
Generations
Design
Feeds
Study A same-window sweep
8 configs × 56 × 2 contracts
one randomized window, effort unset
Tables 4, 15
Study A ladder sweep
44 rungs, 11 configs
per-provider windows
Tables 2, 12
Study A rung crossover
30 rungs, 7 configs, 26 crossover pts
replay both transports
Table 14, Fig. 3
Public three-draw repeat
8 configs × 56 × 2 × 2 draws
same design as the sweep
Appendix E.1.6
Study B native tool
8,736 generations, 17,472 replays
observational, both userlands
Tables 3, 17, 18
Private-v2 crossover
2 models × 42 hostile payloads
one serving window
Table 5
Private-v1 replay
3 models × 42 tasks
earlier generations; script bypass
Table 6
BSD-live legacy
6 configurations
BSD-elicited, GNU-replayed
Table 16
Real-ssh grounding
8 configs × 56
ssh localhost replay
Table 9
Advice arm
8 configs × 56 × 2
same-window paired advice contrast
Table 19
Grammar crossover
8 configs × 56 × 2 disclosed
replay-only wrapper 2×2
Table 20
Real-ssh full crossover
6 configs × 56 × 2
disclosed replies on real ssh
Table 21
JSON boundary
6 configs × 56
serializer replay
Table 22
Table 11: Calibration of the unset-effort sweep arm against each configuration’s labelled ladder. Distance is unset success minus the lowest-rung success, in percentage points. The two measurements come from different serving windows, so small differences are descriptive only.
Model
Unset (%)
Nearest rung
Distance to lowest rung
GPT-5.6-sol
91.1
low
0.0
GPT-5.5
89.3
high
−5.4
Opus-5
89.3
medium
+28.6
Gemini-3.1-Pro
80.4
none within the ladder
−10.7
Gemini-3.5-Flash
58.9
low
+1.8
Opus-4.8
57.1
xhigh
+17.9
Gemini-3.1-Flash-Lite
14.3
all four rungs tie
0.0
Table 12: Matched-nested effort ladders under the disclosed-boundary contract. Within each row, success rates and mean provider-reported output tokens follow the setting order in the second column. Each point uses trial 0 over 56 tasks.
Model
Settings (in order)
Success (%)
Mean output tokens
GPT-5.6-sol
low / medium / high / xhigh
91.1 / 92.9 / 98.2 / 98.2
362 / 565 / 773 / 1,073
GPT-5.5
low / medium / high / xhigh
94.6 / 92.9 / 91.1 / 100.0
507 / 655 / 1,164 / 2,757
Opus-5
low / medium / high / xhigh / max
60.7 / 91.1 / 96.4 / 100.0 / 98.2
796 / 1,336 / 1,458 / 2,421 / 3,499
Fable-5
low / medium / high / xhigh / max
71.4 / 75.0 / 96.4 / 92.9 / 100.0
332 / 569 / 843 / 1,212 / 2,396
Opus-4.8
low / medium / high / xhigh / max
39.3 / 48.2 / 50.0 / 62.5 / 94.6
835 / 1,291 / 1,706 / 3,481 / 6,960
Gemini-3.1-Pro
low / medium / high
91.1 / 91.1 / 91.1
5,267 / 5,753 / 6,308
Sonnet-4.6
low / medium / high / xhigh
23.2 / 28.6 / 64.3 / 62.5
28 / 1,411 / 4,337 / 4,539
Gemini-3.5-Flash
minimal / low / medium / high
57.1 / 58.9 / 64.3 / 62.5
3,851 / 3,108 / 3,081 / 3,389
Haiku-4.5
low / medium / high / xhigh
32.1 / 37.5 / 32.1 / 26.8
5,432 / 4,717 / 4,708 / 4,783
Qwen3.5-27B
non-thinking / thinking
28.6 / 28.6
20 / 7,489
Gemini-3.1-Flash-Lite
minimal / low / medium / high
14.3 / 14.3 / 14.3 / 14.3
20 / 20 / 20 / 20
Table 13: Study-A model identifiers and request parameters. The effort column lists exactly the settings queried; sweep arms omitted the effort field. Temperature is reported only where the interface accepts it.
Display name
Model identifier
Effort settings queried
Decoding parameters
GPT-5.6-sol
gpt-5.6-sol
low, medium, high, xhigh
max output tokens 16,000, temperature not sent
GPT-5.5
gpt-5.5
low, medium, high, xhigh
max output tokens 16,000, temperature not sent
Opus-5
claude-opus-5
low, medium, high, xhigh, max
provider defaults, no sampling or length control sent
Opus-4.8
claude-opus-4-8
low, medium, high, xhigh, max
provider defaults, no sampling or length control sent
Fable-5
claude-fable-5
low, medium, high, xhigh, max
provider defaults, no sampling or length control sent
Sonnet-4.6
claude-sonnet-4-6
low, medium, high, xhigh
provider defaults, no sampling or length control sent
Haiku-4.5
claude-haiku-4-5
low, medium, high, xhigh
provider defaults, no sampling or length control sent
Gemini-3.1-Pro
gemini-3.1-pro-preview
low, medium, high
temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.5-Flash
gemini-3.5-flash
minimal, low, medium, high
temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.1-Flash-Lite
gemini-3.1-flash-lite-preview
minimal, low, medium, high
temperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Qwen3.5-27B
Qwen/Qwen3.5-27B
non-thinking, thinking
temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-9B
Qwen/Qwen3.5-9B
non-thinking, thinking
temperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-4B
Qwen/Qwen3.5-4B
non-thinking, thinking
temperature 0, max tokens 4,096 non-thinking, omitted thinking
Table 14: Crossover at every measured rung. Cells are percentages and effects are percentage points, defined as in Table 4. An asterisk marks the descriptive masked-fragility rule. Gemini-3.1-Flash-Lite returns byte-identical replies at all four settings and is retained only to document that the provider did not expose a usable ladder.
Configuration
Rung
RR
RN
NR
NN
Damage
Compensation
Matched gap
GPT-5.6-sol
low
98.2
28.6
53.6
91.1
−69.6
+62.5
−7.1
GPT-5.6-sol
medium
94.6
30.4
48.2
92.9
−64.3
+62.5
−1.8∗
GPT-5.6-sol
high
98.2
32.1
48.2
98.2
−66.1
+66.1
+0.0∗
GPT-5.6-sol
xhigh
98.2
30.4
51.8
98.2
−67.9
+67.9
+0.0∗
Opus-5
low
98.2
28.6
57.1
60.7
−69.6
+32.1
−37.5
Opus-5
medium
100.0
32.1
41.1
91.1
−67.9
+58.9
−8.9
Opus-5
high
96.4
30.4
42.9
96.4
−66.1
+66.1
+0.0∗
Opus-5
xhigh
98.2
32.1
42.9
100.0
−66.1
+67.9
+1.8∗
Opus-5
max
100.0
32.1
46.4
98.2
−67.9
+66.1
−1.8∗
Gemini-3.1-Pro
low
100.0
25.0
32.1
91.1
−75.0
+66.1
−8.9
Gemini-3.1-Pro
medium
98.2
26.8
41.1
91.1
−71.4
+64.3
−7.1
Gemini-3.1-Pro
high
98.2
26.8
35.7
91.1
−71.4
+64.3
−7.1
Fable-5
low
100.0
30.4
48.2
71.4
−69.6
+41.1
−28.6
Fable-5
medium
100.0
30.4
48.2
75.0
−69.6
+44.6
−25.0
Fable-5
high
98.2
32.1
42.9
96.4
−66.1
+64.3
−1.8∗
Fable-5
xhigh
100.0
33.9
46.4
92.9
−66.1
+58.9
−7.1
Fable-5
max
100.0
30.4
35.7
100.0
−69.6
+69.6
+0.0∗
Gemini-3.5-Flash
minimal
96.4
23.2
75.0
57.1
−73.2
+33.9
−39.3
Gemini-3.5-Flash
low
92.9
25.0
67.9
58.9
−67.9
+33.9
−33.9
Gemini-3.5-Flash
medium
96.4
23.2
76.8
64.3
−73.2
+41.1
−32.1
Gemini-3.5-Flash
high
96.4
23.2
71.4
62.5
−73.2
+39.3
−33.9
Opus-4.8
low
87.5
28.6
69.6
39.3
−58.9
+10.7
−48.2
Opus-4.8
medium
92.9
28.6
67.9
48.2
−64.3
+19.6
−44.6
Opus-4.8
high
96.4
33.9
64.3
50.0
−62.5
+16.1
−46.4
Opus-4.8
xhigh
98.2
32.1
60.7
62.5
−66.1
+30.4
−35.7
Opus-4.8
max
98.2
30.4
44.6
94.6
−67.9
+64.3
−3.6∗
Gemini-3.1-Flash-Lite
minimal
78.6
19.6
80.4
14.3
−58.9
−5.4
−64.3
Gemini-3.1-Flash-Lite
low
78.6
19.6
80.4
14.3
−58.9
−5.4
−64.3
Gemini-3.1-Flash-Lite
medium
78.6
19.6
80.4
14.3
−58.9
−5.4
−64.3
Gemini-3.1-Flash-Lite
high
78.6
19.6
80.4
14.3
−58.9
−5.4
−64.3
Table 15: Enumerated and Holm-adjusted two-sided family-sign p values for the two primary Study-A components under the single-clause disclosed-boundary contract. Effect sizes are percentage points.
Fixed-reply transport
Contract-conditioned compensation
Model
Effect [95% CI]
Enum. p
Holm p
Effect [95% CI]
Enum. p
Holm p
GPT-5.6-sol
−64.3 [−80.4,−46.4]
.000244
.001465
+60.7 [+46.4,+75.0]
.000244
.001953
GPT-5.5
−71.4 [−85.7,−55.4]
.000244
.001465
+60.7 [+44.6,+75.0]
.000244
.001953
Opus-5
−66.1 [−82.1,−50.0]
.000244
.001465
+58.9 [+41.1,+75.0]
.000488
.002930
Gemini-3.1-Pro
−73.2 [−87.5,−58.9]
.000122
.000977
+55.4 [+33.9,+73.2]
.001221
.006104
Gemini-3.5-Flash
−67.9 [−82.1,−51.8]
.000244
.001465
+30.4 [+12.5,+48.2]
.013672
.041016
Opus-4.8
−64.3 [−80.4,−46.4]
.000244
.001465
+30.4 [+14.3,+48.2]
.003906
.015625
Qwen3.5-27B
−55.4 [−69.6,−41.1]
.000244
.001465
0.0 [0.0,0.0]
1.000000
1.000000
Gemini-3.1-Flash-Lite
−58.9 [−71.4,−48.2]
.000122
.000977
−5.4 [−10.7,0.0]
.250000
.500000
Table 16: Cross-userland transfer in the earlier BSD-live campaign. Each cell reports pass rate on BSD live execution and GNU replay of the identical stored command. Arms retain their original campaign multiplicity: raw cells contain 56 records, while nested cells contain 56 or 168 depending on the configuration. We interpret only the within-arm BSD→GNU change. The final column summarizes the higher-transfer userland; it is not a counterfactual GNU-targeted generation result.
Model
Raw BSD→GNU
Nested BSD→GNU
Transfers better to
Fable-5
→91.1
→87.5
BSD
Qwen3.5-27B (non-thinking)
→87.5
→32.1
GNU
Qwen3.5-27B (thinking)
→82.1
→50.0
GNU
Gemini-3.1-Pro
→100.0
→96.4
GNU
Gemini-3.5-Flash
→96.4
→67.9
BSD
Opus-4.8
→87.5
→73.8
BSD (raw); tie (nested)
Table 17: Exploratory native-minus-raw effects over each measured effort ladder, ordered by GNU replay effect.
BSD live
GNU replay
Model
Δ
LOFO
PF/FP
Δ
LOFO
PF/FP
Opus-4.8
+3.10
[+2.18,+3.46]
5/31
+2.62
[+1.67,+2.95]
11/33
Opus-5
−0.36
[−0.51,0.00]
12/9
−0.83
[−1.03,−0.51]
16/9
Fable-5
+0.95
[0.00,+1.15]
3/11
−2.14
[−2.44,−0.64]
22/4
GPT-5.6-sol
−1.19
[−1.92,−0.48]
25/17
−2.53
[−3.37,−0.64]
26/9
Gemini-3.1-Pro
+1.98
[−0.64,+3.21]
8/18
−3.77
[−4.91,−0.64]
23/4
Gemini-3.5-Flash
−4.02
[−4.33,−3.04]
37/10
−9.97
[−10.74,−6.89]
77/10
Table 18: Study-B execution outcomes, aggregated across six models. Each row partitions 4,368 executions. Adherence denotes an invalid one-call tool invocation. Syntax includes parser and command-usage errors. Exit-0 wrong is a silent final-state failure.
Userland
Contract
Pass
Adherence
Syntax
Nonzero
Exit-0 wrong
BSD
raw
4230
0
31
59
48
BSD
native
4236
20
9
41
62
GNU
raw
4252
0
39
36
41
GNU
native
4146
20
22
128
52
Table 19: Matched nested success (of 56) under the disclosed contract without and with escaping advice, paired within one serving window. Advice barely moves the top of the ladder, moves the middle of the ladder the most, and does not reach the bottom.
Configuration
Advice-free
With advice
Δ (pp)
GPT-5.6-sol
54
49
−8.9
GPT-5.5
49
53
+7.1
Opus-5
52
54
+3.6
Sonnet-4.6
32
46
+25.0
Haiku-4.5
25
32
+12.5
Opus-4.8
30
34
+7.1
Qwen3.5-27B
17
18
+1.8
Gemini-3.1-Flash-Lite
7
8
+1.8
Table 20: Grammar crossover (tasks passed of 56). Rows are the disclosed grammar, columns the executed wrapper; the diagonal is matched, the anti-diagonal mismatched. The diagonal advantage separates the same top, middle, and bottom groups as the matched-nested scores.
Double disclosed
Single disclosed
Diag−anti
Configuration
on double
on single
on single
on double
(pp)
GPT-5.6-sol
54
7
53
10
+80.4
GPT-5.5
49
8
54
8
+77.7
Opus-5
52
8
45
16
+65.2
Sonnet-4.6
32
11
23
16
+25.0
Opus-4.8
30
11
23
18
+21.4
Haiku-4.5
25
13
20
11
+18.8
Qwen3.5-27B
17
11
11
17
+0.0
Gemini-3.1-Flash-Lite
7
19
9
19
−19.6
Table 21: Real-ssh full crossover (tasks passed of 56). Damage is RN−RR and compensation is NN−RN, both on the real ssh path. The final column repeats the synthetic nested compensation for comparison.
Configuration
RR
RN
NR
NN
ssh comp.
Nested comp.
GPT-5.6-sol
53
17
31
51
+60.7
+60.7
GPT-5.5
56
16
28
50
+60.7
+60.7
Opus-5
54
17
24
50
+58.9
+58.9
Opus-4.8
51
15
35
32
+30.4
+30.4
Qwen3.5-27B
48
17
47
17
+0.0
+0.0
Gemini-3.1-Flash-Lite
44
11
45
10
−1.8
−5.4
Table 22: JSON serializer boundary (tasks passed of 56). Correct-serializer damage is ≈0; naive-embedding damage is comparable to the shell nested transport. Unparseable counts replies whose naive JSON embedding fails to parse.
Configuration
Raw
Correct
Naive
Naive damage
GPT-5.6-sol
53
53
21
−57.1
GPT-5.5
56
56
19
−66.1
Opus-5
54
54
22
−57.1
Opus-4.8
51
51
18
−58.9
Qwen3.5-27B
48
48
19
−51.8
Gemini-3.1-Flash-Lite
44
44
8
−64.3
Why it matters
In real deployments, agents routinely send commands over ssh, into containers, or through CI pipelines, which is exactly the re-parsing boundary this paper studies, so a leaderboard that ignores it can misrepresent how a model performs once actually deployed. Anyone evaluating or shipping a command-issuing coding agent should check which execution path and generation contract a reported score assumes, not just the score itself.
Terms in this paper
matched score · the standard reported success rate obtained by running a model's generated command directly, without decomposing where failures come from
generation contract · the instruction that tells a model how it is expected to format or express its command output
execution transport · the path a generated command travels before it reaches a real shell, e.g. run directly or re-wrapped inside another parser
nested transport · a test condition that adds one extra double-quoted parsing layer, mimicking how ssh, container, or CI wrappers re-parse a command
final-state validation · scoring based only on whether the resulting files, arguments, or repository state exactly match the expected outcome, regardless of how the command got there
Original abstract (English)
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.
