Figure 1: T4 six-condition coordination ablation: AgentRoom is the top condition, while concurrency without coordination (parallel-merge) and sequential hand-off (ChatDev-style) fall below the single-agent baseline.
Table 1: T4 Sonnet 4.6 six-condition ablation (LLM-judge composite, budget-fair 30–700 s pool). Parallel-merge vs AgentRoom: +0.213, Welch t=3.35, 𝐩=0.003. The shared+collab-noMCP probe ran under a 1200 s wall budget; its runs are filtered to the same 30–700 s envelope as every other cell (Appendix B.13). Cross-model replication in Appendix B, Table 8.
Condition
Mean
n
σ
Note
ChatDev-style (sequential 3-phase)
0.333
6
0.197
sequential paradigm baseline
Parallel merge (2 indep + union)
0.456
12
0.178
concurrent, no coordination
Solo (1 agent)
0.544
32
0.222
single-agent baseline
Shared only (2 agents, no MCP)
0.575
11
0.155
shared CRDT, no signalling
Shared + collab prompt, noMCP
0.588
7
0.188
+ prompt, no MCP tools
AgentRoom (2 agents, full)
0.669
14
0.140
CRDT + room MCP + collab prompt
Figure 2: AgentRoom architecture: N coding agents share a workspace mediated by a CRDT; the MCP server exposes claim, release, broadcast, state, read.
Table 2: Cross-model T4 ES with bootstrap 95% CI (10 000 resamples), AST-scored subsample of the full pool (≥30 s elapsed). n is smaller than the LLM-scored pools because the AST scorer covers a subsample. Headline scorer is AST (TypeScript compiler API); regex-scorer values after the slash for cross-check. ∗ CI excludes 1. Two CLI-unstable models (Gemini, GPT-5.4-mini) excluded (see §4.4). Reported as exploratory conjecture: only Haiku T4 clears the CI under the AST scorer.
Model
SWE-bench V.
Solo (n)
×2 (n)
ES (AST / regex)
95% CI (AST)
Haiku 4.5
73.3%
0.520 (13)
0.929 (8)
1.79× / 1.58×
[1.24, 3.35]∗
Codex GPT-5.4
∼80%
0.248 (9)
0.487 (13)
1.96× / 1.07×
[0.85, 8.85]
Sonnet 4.6
79.6%
0.463 (11)
0.630 (15)
1.36× / 1.14×
[0.84, 2.29]
Figure 3: Real-time CRDT merge during a single T4 Sonnet AgentRoom ×2 run. The MCP-mediated chat (left) drives concurrent character-level edits in the shared file (right); both α’s and β’s insertions co-exist on the same line via the Yrs CRDT, with no byte-level conflict raised in this run. Per-message Bezier leaders link each chat broadcast to the code line it talks about.
Table 3: Sonnet 4.6 ES across the difficulty gradient (regex/AST scorer, outlier-filtered ≥0.3).
Task
Solo (n)
×2 (n)
ES
T1
0.919 (4)
0.910 (7)
0.99×
T2
0.840 (3)
0.904 (4)
1.08×
T3
0.887 (3)
0.943 (4)
1.06×
T4
0.796 (10)
0.958 (10)
1.20×
T5
0.697 (7)
0.810 (7)
1.16×
Figure 4: Solo (slate) vs AgentRoom ×2 (rust) score distributions for the three CLI-stable models on T4 (LLM-judge composite, budget-fair pool). σ contracts by ∼30–45% under AgentRoom in every panel (Sonnet 0.23→0.14, Haiku 0.25→0.17, Codex 0.27→0.15); mean shift varies by model.
Table 4: Heterogeneous vs. homogeneous ×2 on T4 (LLM-judge, budget-fair). Tests is the mean over all budget-fair runs in the cell (n=17/4/3), which is not the LLM-scored n shown in the Score column.
Pair
Score (n)
Tests
Sonnet ×2 (homo)
0.669 (14)
32.8
Sonnet+Codex
0.721 (3)
16.0
Sonnet+Gemini
0.542 (3)
22.0
Figure 5: Agent-count scaling on T4 Sonnet 4.6 (N=1–4, budget-fair pool). Left: mean LLM-judge composite with σ band peaks at N=2 then declines. Middle: mean tests passing peaks at N=3. Right: AgentRoom message count rises super-linearly past N=3, consistent with a single-broadcast-channel coordination overhead.
Table 5: Wall vs compute on T4 Sonnet (LLM-judge composite, budget-fair pool).
Condition
N
Wall (s)
Compute (s)
Quality
Solo
1
550
550
0.544
Parallel merge
2
558
1077
0.456
Shared only
2
601
1201
0.575
AgentRoom ×2
2
561
1072
0.669
AgentRoom ×3
3
566
1572
0.553
Figure 6: Cochran-Mantel-Haenszel forest plot of Solo-vs-AgentRoom 1-file abandonment across all 12 model×task strata (Sonnet/Haiku/Codex × T1/T2/T4/T5).
Table 6: T5 (trading platform, 900 s budget), AST scorer, per-model cells. All CLI-stable cells are point-estimate-positive on ×2 except Codex; CIs do not exclude 1 at this n. The Gemini row is CLI-unstable and informational only (§4.4); sparse heterogeneous cells are in the text above.
Model
Solo (n)
×2 (n)
×3 (n)
ES(×2)
Tests (×2)
Haiku 4.5
0.416 (8)
0.707 (7)
0.822 (1)
1.70×
38
Sonnet 4.6
0.618 (7)
0.674 (7)
0.730 (2)
1.09×
57
GPT-5.4
0.667 (9)
0.531 (7)
0.761 (1)
0.79×
8
Gemini Flash (unstable)
0.851 (3)
0.506 (9)
0.246 (3)
0.59×
12
Figure 7: Bootstrap effect-size distributions (10 000 resamples) for the Tier III cross-model cells. Only Haiku T4 has its 95% CI strictly above 1; the other five cells span 0.79–1.96× point-estimate with CIs that include 1.
Table 7: Frequency of unprompted collaborative behaviors across 6 valid ×2 AgentRoom runs on T4 Sonnet.
Behavior
Runs exhibiting (of 6)
Module claiming via broadcast
6/6
Plan adjustment after reading teammate
5/6
Export list shared for teammate
4/6
Proactive gap filling
3/6
Cross-agent bug fix
2/6
Apology for file boundary violation
2/6
Figure 8: Cost–quality Pareto on T4. Shaded region is Pareto-dominated; dotted lines are iso-efficiency (quality/cost) contours. 2×Haiku AgentRoom clears 1×Sonnet at matched plotted cost.
Table 8: Cross-model T4 ablation (LLM-judge composite, budget-fair pool). For Sonnet and Haiku, AgentRoom is the top condition and parallel-merge falls below solo. Codex non-Solo/AgentRoom cells were not collected (vendor-CLI hang).
Baselines and partial bundles
Ours
Model
Solo
Parallel merge
Shared only
noMCP
AgentRoom ×2
Sonnet 4.6
0.544 (n=32)
0.456 (n=12)
0.575 (n=11)
0.588 (n=7)
0.669 (n=14)
Haiku 4.5
0.530 (n=35)
0.383 (n=12)
0.657 (n=7)
0.579 (n=6)
0.662 (n=17)
Codex GPT-5.4
0.384 (n=12)
–
–
–
0.529 (n=17)
Figure 9: Single representative T4 Sonnet AgentRoom ×2 run. Top swim-lane: agent α broadcasts (timestamps in-card); middle: per-file ownership Gantt (rust = α, slate = β); bottom: agent β. Bezier S-curve leaders link each chat broadcast to its time on the Gantt. The whole protocol (claim, conflict-detect, hand-off, completion broadcast) emerges from the advisory six-step prompt (§2.3) without any orchestrator.
Table 9: Threshold sensitivity for the Tier I CMH. Each row is a threshold combination; cells report Solo/AgentRoom abandonment counts per model and the resulting CMH common OR with 95% CI and p-value.
Thresholds
Sonnet 4.6
Haiku 4.5
Codex GPT-5.4
CMH pooled
d thr
f thr
Solo
Room
Solo
Room
Solo
Room
OR
95% CI
p
<150
≤1
6/33
0/17
12/35
1/17
9/17
0/21
24.2
[3.5, 169]
0.00001
<150
≤2
6/33
0/17
12/35
1/17
9/17
2/21
11.9
[3.0, 47]
0.00005
<150
≤3
6/33
0/17
12/35
1/17
10/17
6/21
6.3
[2.1, 19]
0.00069
<200
≤1
6/33
0/17
12/35
1/17
9/17
0/21
24.2
[3.5, 169]
0.00001
<200
≤2 (paper)
6/33
0/17
12/35
1/17
9/17
2/21
11.9
[3.0, 47]
0.00005
<200
≤3
6/33
0/17
12/35
1/17
10/17
6/21
6.3
[2.1, 19]
0.00069
<250
≤1
6/33
0/17
12/35
1/17
9/17
0/21
24.2
[3.5, 169]
0.00001
<250
≤2
6/33
0/17
12/35
1/17
9/17
2/21
11.9
[3.0, 47]
0.00005
<250
≤3
6/33
0/17
12/35
1/17
10/17
6/21
6.3
[2.1, 19]
0.00069
<300
≤1
6/33
0/17
12/35
1/17
9/17
0/21
24.2
[3.5, 169]
0.00001
<300
≤2
6/33
0/17
12/35
1/17
9/17
2/21
11.9
[3.0, 47]
0.00005
<300
≤3
6/33
0/17
12/35
1/17
10/17
6/21
6.3
[2.1, 19]
0.00069
Figure 10: The two scorers measure the same thing loosely, not interchangeably. Each point is one T4 run carrying both a regex composite and an LLM-judge composite (n=246, the runs scored by both); Pearson r=0.67. The spread off the diagonal is why the body uses one scorer throughout and reports the other as a cross-check rather than averaging them.
Table 10: Pool sensitivity for the T4 Sonnet ablation (LLM-judge composite). The substrate-vs-AgentRoom ordering is preserved under both pools; the budget-fair pool is primary.
Pool
Solo
Parallel merge
Shared only
AgentRoom ×2
30–700 s (paper, budget-fair)
0.544 (n=32)
0.456 (n=12)
0.575 (n=11)
0.669 (n=14)
≥30 s (full pool)
0.464 (n=42)
0.436 (n=13)
0.527 (n=14)
0.639 (n=15)
Table 11: Solo + collaboration-style prompt without teammates, T4, n=3 each (LLM-judge composite). The structured prompt alone is model-dependent: it hurts Sonnet (channel is the load-bearing variable) and helps Haiku (the structured prompt also acts as workflow scaffolding); the channel and prompt-as-workflow are not separately identified for Haiku at this n.
Model
Solo (bare, budget-fair)
Solo + collab prompt (n=3)
AgentRoom ×2 (budget-fair)
Sonnet 4.6
0.544 (n=32)
0.448 (range 0.20–0.62)
0.669 (n=14)
Haiku 4.5
0.530 (n=35)
0.604 (range 0.55–0.66)
0.662 (n=17)
Table 12: DevBench cross-domain validation (Haiku 4.5, 600 s budget). Pass = hidden pytest suite passes; durations in seconds. geotext: ×2 completes within budget while solo times out; stocktrends, textcnn: both fail (task too hard for budget).
Task
Solo (PASS, dur)
×2 AgentRoom (PASS, dur)
arxiv-digest
PASS, 70 s
PASS, 120 s
readtime
PASS, 110 s
PASS, 100 s
chakin
PASS, 130 s
PASS, 130 s
geotext
FAIL, 603 s timeout
PASS, 390 s
hone
PASS, 90 s
PASS, 160 s
hybrid-images
PASS, 100 s
PASS, 170 s
lice
PASS, 330 s
PASS, 270 s
particle-swarm-opt
PASS, 100 s
PASS, 120 s
stocktrends
FAIL, 600 s
FAIL, 480 s
textcnn
FAIL, 601 s
FAIL, 856 s
Pass rate
7/10
8/10
Table 13: Scorer sensitivity: leave-one-out on T4 (Sonnet 4.6, solo vs ×2 AgentRoom). The solo mean reported here (0.786) is computed over a smaller subset (4 runs) than the canonical solo subset used in Table 1; the smaller subset is the intersection of solo runs for which we retained per-dimension intermediate scorer outputs at the time of writing, kept here only as a robustness check. The AgentRoom > solo ordering is preserved under every single-dimension ablation and the ratio stays within 1.12–1.25 on this subset.
Concurrent multi-agent coding promises division of labor across modules, robustness through redundancy, and parallel exploration at the natural granularity of multi-file projects. Realtime collaborative editing protocols solve this coordination problem for human teams via Conflict-free Replicated Data Types (CRDTs), but the LLMs underneath generate one token at a time and existing multi-agent coding systems inherit this serial limit: they either sequence agents through phase handoffs or pool independent samples without coordination, and a single agent abandons up to half of hard tasks with a one-file stub-and-exit. AgentRoom is a realtime collaborative editing protocol for concurrent coding agents. Its runtime layer exposes file-level claim, status, and broadcast as MCP tools on a CRDT-merged shared filesystem. Five frontier coding-CLI models ran four backend coding tasks, with cross-language checks in Python DevBench and Rust+axum. For CLI-stable models, AgentRoom with 2 agents abandons fewer tasks than Solo and has less run-to-run variation. At matched-compute, one positive mean LLM-judge contrast puts AgentRoom over parallel-merge. The other contrast, a bundle probe, puts full AgentRoom above each partial case: an ordering rather than a percentage split. Coordination, not parallelism or CRDT-merge, bears the load.
