工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

arXiv:2608.024992026-08-02

一个测量编码智能体在任务进行中被用户直接改动代码时会崩到什么程度的基准

SWE-Touch 是一个评测框架,模拟用户在编码智能体执行任务过程中直接修改代码,而不只是发消息。它会在智能体正在查看的代码附近,注入一个看似合理但与任务目标冲突的假用户修改(Counter-Edit),并在 SWE-bench Verified 及两个更长任务的基准上测量九个模型的反应。结果显示平均解决率下降了7.7个百分点,单独编码能力强并不能保证在这种干扰下依然稳定。

METAL LAB 解读图

SWE-Touch 的注入与评测流程

证据状态已报告实测结果

  1. 挖掘关键代码区域让三个不同模型(GPT 5.5、GLM 5.1、MiniMax M2.7)各自完成任务,取其轨迹重叠部分找出对解题至关重要的代码区域。
  2. 生成并验证 Counter-Edit独立的 User Patch Generator 在这些区域附近构造一个看似合理但与任务冲突的修改,并验证它单独或与参考修复合并都无法解决任务。
  3. 注入共享工作区只要智能体的动作触及目标区域,系统就把这个假修改和带上下文的用户消息注入实时代码库,默认最多注入三次。
  4. 验证结果并归类失败原因用测试检验最终代码,对失败案例按是否保留冲突代码、错误替换、还是不完整调和等类型进行分类。
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 现有的编码智能体基准大多让智能体独自工作,或只允许用户通过消息参与,但作者发现已发布的 SWE-chat 对话数据中,有59.0%的会话实际包含用户直接修改代码库的记录。
  2. SWE-Touch 让三个不同模型(GPT 5.5、GLM 5.1、MiniMax M2.7)各自完成同一任务,再取它们轨迹的重叠部分找出任务关键代码区域,然后用一个独立的 User Patch Generator 在这些区域附近构造一个看似合理但会阻碍任务完成的代码修改。
  3. 每个候选修改都经过三项验证:假用户修改单独不能解决任务,参考修复方案能解决任务,而两者合并后仍然无法解决——以确保冲突是真实且非平凡的。
  4. 评测时,一旦智能体的动作触及目标代码区域,系统就会将这个假修改和一条带上下文的用户消息注入代码库,默认最多注入三次,随后观察智能体如何继续应对并用测试检验最终结果。
  5. 在 SWE-bench Verified 上对九个模型的评测显示,Counter-Edit 条件下平均解决率下降了7.7个百分点,在更长任务的 SWE-Bench Pro 和 DeepSWE 基准上性能下降依然持续。
Figure 1: Users and agents share a workspace in real coding sessions. (a) A user edit alters the repository state that subsequent agent actions observe and modify. (b) Our analysis of the released SWE-chat data [5] finds that 59.0% of sessions contain repository changes attributed to the user.
Figure 1: Users and agents share a workspace in real coding sessions. (a) A user edit alters the repository state that subsequent agent actions observe and modify. (b) Our analysis of the released SWE-chat data [5] finds that 59.0% of sessions contain repository changes attributed to the user.
Table 1: Comparison of user interaction modes in coding-agent benchmarks.
BenchmarkCodeUserUserCodebase
RepairSimulatorMessagesEdit
SWE-bench Verified✓✗✗✗
Ambig-SWE✓✓✓✗
HiL-Bench✓✓✓✗
SWE-Interact✓✓✓✗
SWE-Together✓✓✓✗
SWE-Touch (ours)✓✓✓✓
Figure 2: Overview of SWE-Touch. Agent trajectories identify task-critical regions; a separate generator constructs and validates task-conflicting edits; evaluation injects each edit and its contextual message into the shared workspace before task verification.
Figure 2: Overview of SWE-Touch. Agent trajectories identify task-critical regions; a separate generator constructs and validates task-conflicting edits; evaluation injects each edit and its contextual message into the shared workspace before task verification.
Table 2: Average patch size, shown as changed lines / files.
SourceReference repairCounter-Edit
Lines / filesLines / files
SWE-bench Verified13.3 / 1.207.0 / 1.04
SWE-Bench Pro361.0 / 5.4413.0 / 1.40
DeepSWE730.2 / 7.2410.8 / 1.52
Figure 3: Longer-horizon robustness under user edits on SWE-Bench Pro and DeepSWE: (a) cost–performance shifts and (b) sensitivity to edit frequency.
Figure 3: Longer-horizon robustness under user edits on SWE-Bench Pro and DeepSWE: (a) cost–performance shifts and (b) sensitivity to edit frequency.
Table 3: Main results on SWE-bench Verified over three runs. Resolve is mean ± standard deviation; Steps and Tok. (K) are completed-task means, with tokens reported in thousands. Δ is Counter-Edit minus Vanilla; Retention is the share of majority-solved Vanilla tasks that remain majority-solved under Counter-Edit. (1) indicates mean resolve-rate ranking; Rank Δ shows the corresponding ranking change.
VanillaCounter-EditΔRetentionRank
ModelResolveStepsTok. (K)ResolveStepsTok. (K)ResolveSteps(%)Δ
Claude Opus 4.885.2±1.8 (1)24.536783.3±0.6 (1)30.9506-1.8+6.496.0–
GPT 5.580.5±1.0 (2)33.21,34479.2±0.6 (2)31.31,141-1.3-1.995.0–
GLM 5.172.7±2.0 (7)55.01,00768.3±0.8 (4)64.51,272-4.3+9.583.3↑3
MiniMax M2.776.5±1.5 (3)45.386562.7±2.4 (8)47.7904-13.8+2.478.1↓5
MiniMax M2.575.7±3.3 (4)45.186366.2±1.0 (5)47.5908-9.5+2.478.3↓1
Qwen 3.7 Max75.2±1.0 (5)29.942070.3±0.8 (3)31.1424-4.8+1.290.3↑2
Qwen3-Coder-480B57.2±3.5 (9)52.676340.7±1.0 (9)54.6806-16.5+2.060.8–
Kimi K2.670.3±2.0 (8)62.51,38164.3±3.4 (6)62.11,296-6.0-0.487.2↑2
DeepSeek V4 Pro74.8±0.8 (6)41.982763.8±1.8 (7)46.6954-11.0+4.781.5↓1
(b) Resolve rate under varying edit frequency (K) on two longer-horizon benchmarks.
(b) Resolve rate under varying edit frequency (K) on two longer-horizon benchmarks.
Table 4: Mean resolve rates on SWE-Bench Pro and DeepSWE over two runs. Δ Res. is the change in resolve rate, and Δ Steps is the change in mean model calls per observed trajectory. Both differences are Counter-Edit minus Vanilla.
SWE-Bench ProDeepSWE
ModelVanillaCounter-EditΔ Res.Δ StepsVanillaCounter-EditΔ Res.Δ Steps
Claude Opus 4.868.068.00.0+7.556.046.0-10.0+9.9
GPT 5.538.038.00.0+0.964.056.0-8.0+4.9
GLM 5.143.132.8-10.3+19.119.416.8-2.5+31.4
MiniMax M2.730.624.6-6.0+9.42.22.20.0+27.6
MiniMax M2.532.624.6-8.0+20.40.00.00.0+33.1
Qwen 3.7 Max36.026.0-10.0−1.44.12.1-2.0+2.3
Qwen3-Coder-480B20.014.0-6.0+0.70.00.00.0+7.8
Kimi K2.650.048.0-2.0−4.718.012.0-6.0+0.2
DeepSeek V4 Pro34.032.0-2.0+8.94.12.0-2.1−7.6
Figure 4: Failure analysis of solved-to-unresolved runs across nine models on SWE-bench Verified. (a) Overall distribution across seven failure types. (b) Per-model composition. (c) Percentage of failures in which the agent revises or removes the user edit before termination. Sample sizes and audit details are reported in Appendix D.
Figure 4: Failure analysis of solved-to-unresolved runs across nine models on SWE-bench Verified. (a) Overall distribution across seven failure types. (b) Per-model composition. (c) Percentage of failures in which the agent revises or removes the user edit before termination. Sample sizes and audit details are reported in Appendix D.
Table 5: Resolve rates for message, code-edit, and edit-frequency ablations on SWE-bench Verified. Δ is relative to the paired Vanilla run.
InterventionGPT 5.5GLM 5.1MiniMax M2.7Qwen 3.7 Max
ResolveΔResolveΔResolveΔResolveΔ
Vanilla81.5–70.5–76.5–74.0–
Message (K=3)79.5-2.073.0+2.576.50.077.0+3.0
Code edit (K=3)80.5-1.066.5-4.067.0-9.571.5-2.5
Both (K=1)78.5-3.072.0+1.564.5-12.071.5-2.5
Both (K=3)79.5-2.069.0-1.564.5-12.071.0-3.0
Both (K=5)78.0-3.569.0-1.560.0-16.569.0-5.0
Figure 5: Post-edit behavior on a diagnostic sample from SWE-bench Verified. (a) Mean read, edit, and test commands issued after the final user edit. (b) Agent response mode to the user edit.
Figure 5: Post-edit behavior on a diagnostic sample from SWE-bench Verified. (a) Mean read, edit, and test commands issued after the final user edit. (b) Agent response mode to the user edit.
Table 7: List prices used to convert tokens into dollars in Figure 3(a), as displayed by the serving endpoint on 27 July 2026; OpenRouter rates are the promotional prices shown at that time. Cached input is charged at the input rate.
ModelEndpointIn ($/M)Out ($/M)
Claude Opus 4.8Anthropic5.0025.00
GPT 5.5OpenAI5.0030.00
GLM 5.1OpenRouter0.9663.036
MiniMax M2.7OpenRouter0.240.96
MiniMax M2.5OpenRouter0.150.90
Qwen 3.7 MaxOpenRouter1.4754.425
Qwen3-Coder-480BOpenRouter0.221.80
Kimi K2.6OpenRouter0.6462.72
DeepSeek V4 ProOpenRouter0.4350.87
Figure 6: Task-level outcome transitions from Vanilla to Counter-Edit on SWE-bench Verified. Arrows indicate the direction and magnitude of shifts between solved and unresolved states.
Figure 6: Task-level outcome transitions from Vanilla to Counter-Edit on SWE-bench Verified. Arrows indicate the direction and magnitude of shifts between solved and unresolved states.
Table 10: Verifier outcomes under Vanilla and Counter-Edit, by model.
Van. solvedVan. unresolved
Modelkept→unres.→solvedkept
GPT 5.51528733
Claude 4.81667324
GLM 5.1125251634
MiniMax M2.712134639
MiniMax M2.5119331038
Qwen 3.7 Max13915442
Qwen3-Coder7347971
Kimi K2.612318950
DeepSeek V412328742

研究结果

  • 九个模型在 SWE-bench Verified 上的平均解决率,在 Counter-Edit 条件下比独立作业(Vanilla)低7.7个百分点,各模型间损失幅度从1.3到16.5个百分点不等。
  • 独立作业表现最强的两个模型 Claude Opus 4.8(85.2%→83.3%)和 GPT 5.5(80.5%→79.2%)几乎保持稳定且排名不变,而 Vanilla 分数相近的中游模型在 Counter-Edit 后排名剧烈变化,例如 MiniMax M2.7 从第3名跌到第8名。
  • 对之前已解决但后来失败的运行进行审查发现,63.3%的失败仍保留了用户的冲突代码未处理,13.9%用另一个错误实现替换了它,11.6%只做了不完整的调和。
  • 只发送用户消息而不改代码,影响很小且各模型表现不一致(-2.0到+3.0个百分点);但悄悄只改代码不发消息,则每个模型都出现持续下降(-1.0到-9.5个百分点)。
  • 作为对照的无害修改(Co-Edit,不与任务冲突)几乎不影响平均解决率(仅-0.1个百分点),说明问题的核心在于代码本身与任务目标的语义冲突,而不只是出现了外部修改这件事本身。

可应用场景

  • 该框架可用于压力测试面向结对编程或实时代码审查场景的编码助手,检验其在用户可能同时修改代码时的稳健性。
  • 可作为一种检查手段,评估智能体在外部代码变更后是否养成了重新核查和重新测试的习惯,而不是盲目沿用之前的计划。
  • 团队在选择编码智能体时,可以将共享工作区下的稳健性作为独立的评价维度,而不仅仅依赖独立自主的排行榜分数。

局限与待验证事项

  • 结果基于特定抽样的任务集(SWE-bench Verified 200个任务,SWE-Bench Pro 和 DeepSWE 各25个任务),尚未验证在其他代码库、编程语言或任务类型上的普适性。
  • 模拟的用户消息由 GPT-4o 生成,而非真实人类撰写,可能无法完全反映真实用户表达修改意图的多样方式。
  • Counter-Edit 是刻意构造出的、明确与任务冲突的极端情形,真实用户的修改未必总是如此清晰对立。
  • 在两个更长任务的基准上,编辑触发方式改为按轨迹进度比例注入,而非按代码区域重叠触发,因此这些结果与主实验设置不能直接比较。
  • 论文指出了检测变化、调和冲突、重新验证这三项所需能力,但尚未提出或测试一种能可靠同时满足这三点的智能体设计方案。

为什么重要

如果编码智能体无法承受人类在共享工作区中直接修改代码,这对真实的结对编程和协作开发场景是一个严重缺口,而不只是学术上的好奇。这项研究表明,静态排行榜式的独立表现和在共享工作区协作中的稳健性是两个不同的维度,这对于选择在真实协作编码场景中信任哪个智能体的人很有参考价值。

本文术语

  • Counter-Edit · 一种看似合理但实际错误、被设计用来与任务完成目标冲突的假用户代码修改
  • resolve rate(解决率) · 通过全部验证测试的任务所占的比例
  • retention(保持率) · 在独立作业(Vanilla)下多数轮次判定为已解决的任务,在引入 Counter-Edit 后仍保持已解决状态的比例
  • User Patch Generator · 专门负责构造并验证与任务冲突的假用户修改的独立智能体
  • 共享工作区(shared workspace) · 用户和智能体共同操作同一个代码库、同一批文件、同一个可执行状态的场景

论文原文摘要（英文）

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to

作者 · Yuqiao Tan

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Yuqiao Tan et al., arXiv:2608.02499, arxiv-nonexclusive