每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

arXiv:2608.197992026-08-21

让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对

研究团队构建了一个名为SWE-bench Science的新测试集,包含从化学、生物、物理等20个科学领域、98个真实GitHub代码仓库中挑出的119个修复任务。即便表现最好的AI编程助手Claude Code配合Opus-5,在严格的隐藏测试下正确修复率也不到50%。研究还发现,给AI额外补充科学背景知识并不总是有帮助,有时反而会拉低准确率。

他们做了什么

  1. 以往的编程测评大多是写简单函数或修一般软件的bug,这次测的是AI能否修复真正的科研代码,比如模拟程序或仪器数据处理流程,同时还要保证科学结论不出错
  2. 任务分成三类:issue驱动型(修复已知bug)、专家探索型(像科学家一样自己找出未知原因)、工程集成型(打通多个模块完成完整功能链)
  3. 测试了GPT-5.6-sol、Claude-Opus-5、DeepSeek-V4-Pro等8种AI配置,它们在公开可见的测试上大多能拿到90%以上的分数,但换成隐藏的严格私有测试后,正确率明显大幅下降
  4. 研究者把失败原因归纳为四类:缺乏科学知识或抽象能力、只做表面修复没找到真正病因、修复不完整导致系统其他部分出问题、无法把科学原理推广到没见过的新情况
  5. 在补充科学背景说明的对比实验中,DeepSeek-V4-flash的表现变好了,但GPT-5.6-sol的精确成功率反而下降,说明信息给得多不代表修复得更好
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? figure 0
Table 4 Distribution of the 119 SWE-bench Science tasks across 20 scientific domains.
Scientific domainTasks
Chemistry24
Materials Science and Engineering16
Biology13
Biomedical Engineering12
Physics11
Mathematics7
Astronomy7
Atmospheric Science5
Civil Engineering5
Surveying and Mapping Science and Technology3
Geophysics3
Mechanics3
Electrical Engineering2
Marine Science2
Geography1
Aeronautical and Astronautical Science and Technology1
Nuclear Science and Technology1
Computer Science and Technology1
Statistics1
Information and Communication Engineering1
Total119
Figure 1 Pass@1 comparison of coding agents on SWE-bench Science.
Figure 1 Pass@1 comparison of coding agents on SWE-bench Science.

为什么重要

随着AI越来越多地参与撰写和修改支撑科研结论的代码,一次有缺陷的修改可能悄悄损害的不只是程序运行,还有科学结论本身的可信度。这项研究用具体证据表明,现在的AI编程助手常常能骗过表面检查却漏掉深层的科学正确性,这对做科研软件开发或打造AI编程工具的人都很关键。

Figure 3 Evaluation pipeline in SWE-bench Science for an example task. The diagram shows the frozen agent-visible inputs, the agent-driven coding loop, and the clean evaluation phase with separate public diagnostics and private scientific cases.
Figure 3 Evaluation pipeline in SWE-bench Science for an example task. The diagram shows the frozen agent-visible inputs, the agent-driven coding loop, and the clean evaluation phase with separate public diagnostics and private scientific cases.

本文术语

  • pass@1 · 一种严格的成功率指标,只有一次尝试就通过全部隐藏私有测试才算成功
  • issue驱动型任务 · 基于真实报告过的bug构建的任务,要求AI直接修复
  • 专家探索型任务 · 原因未知,需要AI像科学家一样自主调查根本原因的任务
  • 工程集成型任务 · 需要打通多个文件和模块、完成一整套功能的任务
  • 公开/私有测试 · 公开测试是AI调试时能看到的检查项,私有测试是提交后才启用的隐藏严格检查,用来判断是否真正修复成功
Figure 4 Combined overview of raw-task construction and class-specific task re-design.
Figure 4 Combined overview of raw-task construction and class-specific task re-design.

无法转载的图表

  • Figure 1
在原文中查看图表 →

论文原文摘要（英文）

Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.

作者 · Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Zhipeng Xu et al., arXiv:2608.19799, CC BY 4.0