SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
Researchers built a new test called SWE-bench Science, made of 119 repair tasks pulled from 98 real GitHub repositories across 20 scientific fields like chemistry, biology, and physics. Even the top-performing AI coding agent, Claude Code paired with Opus-5, solved fewer than half the tasks correctly under strict private testing. The team also found that feeding agents extra scientific background information doesn't always help, and can sometimes hurt accuracy.
What they did
- Unlike older coding benchmarks that test writing short functions or fixing generic software bugs, this one tests whether AI agents can repair real scientific research code, like simulations or instrument data pipelines, while preserving scientific correctness
- Tasks are split into three types: Issue-driven (fix a known bug), Expert-exploratory (figure out an unknown root cause like a scientist would), and Engineering-integration (connect multiple modules into a working pipeline)
- Eight AI agent setups were tested, including GPT-5.6-sol, Claude-Opus-5, and DeepSeek-V4-Pro; they scored well over 90% on visible public checks but their success rate dropped sharply on hidden, stricter private tests
- Failures were grouped into four recurring patterns: lacking scientific knowledge, shallow fixes that only patch symptoms, incomplete fixes that miss system-wide effects, and failing to generalize a scientific principle beyond the exact case shown
- In a follow-up test, giving agents extra scientific explanations helped one model (DeepSeek-V4-flash) but actually lowered the exact-success rate of another (GPT-5.6-sol), showing that more information doesn't automatically mean better repairs
| Scientific domain | Tasks |
|---|---|
| Chemistry | 24 |
| Materials Science and Engineering | 16 |
| Biology | 13 |
| Biomedical Engineering | 12 |
| Physics | 11 |
| Mathematics | 7 |
| Astronomy | 7 |
| Atmospheric Science | 5 |
| Civil Engineering | 5 |
| Surveying and Mapping Science and Technology | 3 |
| Geophysics | 3 |
| Mechanics | 3 |
| Electrical Engineering | 2 |
| Marine Science | 2 |
| Geography | 1 |
| Aeronautical and Astronautical Science and Technology | 1 |
| Nuclear Science and Technology | 1 |
| Computer Science and Technology | 1 |
| Statistics | 1 |
| Information and Communication Engineering | 1 |
| Total | 119 |
Why it matters
As AI increasingly writes and fixes the code behind scientific results, a broken patch can quietly corrupt the evidence behind a published finding, not just crash a program. This work gives concrete evidence that current AI agents often pass surface-level checks while missing deeper scientific correctness, which matters for anyone building or relying on AI coding tools in research.
Terms in this paper
- pass@1 · a strict success score counting a task solved only if every hidden private test passes on the first try
- Issue-driven task · a task built from an actual reported bug that the agent must fix
- Expert-exploratory task · a task where the agent must independently investigate an unknown cause, like a scientist debugging a puzzling result
- Engineering-integration task · a task requiring the agent to connect multiple files and modules into a complete working feature
- public/private tests · public tests are visible to the agent for debugging, private tests are hidden and only run after submission to judge true success
Figures we cannot republish
- Figure 1
Original abstract (English)
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Read on arXivLatest papers
- Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System MessagesForcing image-understanding AI to follow hidden system rules quietly wrecks its accuracy, and it collapses even more when users push back
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy CorrectionWhen text input is missing or broken, this AI doesn't guess once and move on—it revises its guess step by step to read emotions more reliably
- Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-EncoderA first-of-its-kind search benchmark and AI model let you find 1C business-software code using Russian-language questions
- Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right call
- Reliable Financial Named Entity Recognition under Domain ShiftAn AI's confidence trained on formal filings turns unreliable once it reads tweets
- When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language ModelsSlipping an irrelevant sentence into a prompt shifts multimodal AI answers in a predictable, formula-like way
- Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured PartitioningA new way to slice time series into meaningful chunks instead of arbitrary equal-length pieces
Latest from METAL LAB
- OpenAI Closes In on Anthropic Again in Enterprise Spending Share
- Meta Unveils First 10 Tasks in WildArtifactBench, a Benchmark for AI Agents
- Musk: "Optimus + Grok will one day handle healthcare for all humanity"
- 35% of Web Pages Published Since ChatGPT Show Signs of AI Authorship
- GPT-Image-2 adds transparent background preview in API
Figures: Zhipeng Xu et al., arXiv:2608.19799, CC BY 4.0