SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

arXiv:2608.098022026-08-09

让AI编程智能体挑战多文件重构任务,结果最强模型也只解决了41.2%

现有代码基准正快速饱和,一项审查还发现SWE-bench Verified中近60%未解决的用例其测试本身存在缺陷。本文提出SWE-Bench ProMax,一个由专家精心整理、覆盖七种编程语言、共170个真实提交的大规模代码重构基准。在两种智能体框架下评测六个前沿模型后,表现最好的模型解决率也仅为41.2%,说明跨文件协同重构对当前AI智能体仍是真正的难题。

METAL LAB 解读图

SWE-Bench ProMax的构建与评测流程

证据状态已报告实测结果

第一阶段:数据收集从星标500以上、采用认可开源许可证、目标语言占比80%以上的仓库中,挖掘2025年1月后含'refactor'关键词的29,782个候选提交
第二阶段:环境验证为每个候选提交构建Docker环境,验证黄金补丁应用后能否通过完整测试套件,失败者被剔除
第三阶段:专家审校过滤复杂度不足的任务,从零重写问题描述,剔除过窄或过宽的测试,最终确定170个实例
模型评测在mini-swe-agent和OpenHands两种框架下测试GPT-5.2、Claude Sonnet 4.6、Gemini-3-Pro、GLM-5、Kimi-K2.5、Qwen3.5,最高解决率为41.2%
失败模式分析发现智能体在大补丁上修改的文件数明显少于所需,且失败尝试消耗的交互轮数远多于成功尝试

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

研究出发点是现有基准(如SWE-bench Verified)难度已经见顶,且测试质量存在缺陷,导致分数不能真实反映智能体的实际能力。
作者从星标数500以上、采用认可开源许可证、且某一目标语言代码占比80%以上的GitHub仓库中,挖掘出29,782个包含'refactor'关键词的候选提交,再经环境验证与专家改写,最终仅保留170个实例。
每个问题描述都从零重写以消除歧义,测试套件也经人工审查,剔除了过于狭窄(会误判正确解为错误)和过于宽泛(检查未声明要求)的测试。
基准覆盖Python、Java、TypeScript、Go、C、C++、Rust七种语言,平均每个任务需要修改11.4个文件、261.6行代码,规模远超现有基准。
在mini-swe-agent和OpenHands两种智能体框架下评测了Claude Sonnet 4.6、GPT-5.2、Gemini-3-Pro、GLM-5、Kimi-K2.5、Qwen3.5六个前沿模型,GPT-5.2以41.2%的解决率位居第一。

Figure 1: Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.

Table 1: Comparison of SWE-Bench ProMax with existing benchmarks. Avg. >5 Files: the average gold patch modifies more than 5 files. Expert Curated: problem descriptions and/or test suites undergo manual expert review or authoring beyond automated collection.
Benchmark	Execution Based	Repo Level	Multi Lingual	Refac- toring	Avg. >5 Files	Expert Curated
HumanEval [5]	✓					✓
MBPP [2]	✓					✓
LiveCodeBench [19]	✓
SWE-bench [21]	✓	✓
Multi-SWE-bench [49]	✓	✓	✓			✓
SWE-PolyBench [33]	✓	✓	✓
SWE-bench Pro [8]	✓	✓				✓
SWE-EVO [23]	✓	✓			✓
Terminal-Bench [26]	✓	✓				✓
RefactorBench [13]	✓	✓		✓		✓
SWE-Refactor [47]	✓	✓		✓
SWE-Bench ProMax (Ours)	✓	✓	✓	✓	✓	✓

Figure 3: Data collection and curation pipeline for SWE-Bench ProMax.

Table 3: Resolve rates (%) of all evaluated models across scaffolds on SWE-Bench ProMax.
	Overall	Avg. Steps	Avg. Cost	Per-Language Resolve Rate
				Py	Java	TS	Go	C	C++	Rust
Mini-SWE-Agent
Proprietary
Gemini-3-Pro	26.5	58.0	$0.60	17.2	15.4	14.3	26.1	50.0	45.5	27.3
Claude Sonnet 4.6	30.6	99.5	$2.32	13.8	19.2	32.1	13.0	65.0	40.9	40.9
GPT-5.2	21.8	25.2	$0.19	17.2	15.4	21.4	13.0	45.0	31.8	13.6
Open-weight
GLM-5	22.9	108.9	$0.10	13.8	7.7	25.0	21.7	50.0	22.7	27.3
Kimi-K2.5	26.5	85.3	$0.37	17.2	23.1	21.4	17.4	60.0	31.8	22.7
Qwen3.5	20.6	155.4	$0.93	17.2	7.7	10.7	13.0	45.0	27.3	31.8
OpenHands
Proprietary
Gemini-3-Pro	19.4	51.2	$1.49	13.8	19.2	0.0	8.7	45.0	36.4	22.7
Claude Sonnet 4.6	38.8	117.9	$4.77	17.2	30.8	53.6	26.1	50.0	36.4	63.6
GPT-5.2	41.2	115.1	$3.60	48.3	19.2	35.7	26.1	75.0	36.4	54.5
Open-weight
GLM-5	36.5	114.2	$0.24	20.7	34.6	28.6	34.8	65.0	45.5	36.4
Kimi-K2.5	32.9	99.6	$0.72	24.1	30.8	10.7	43.5	70.0	45.5	18.2
Qwen3.5	36.5	141.2	$0.78	37.9	26.9	17.9	39.1	65.0	54.5	22.7

Figure 4: Language distribution in SWE-Bench ProMax.

Table 4: Repositories and licenses used in SWE-Bench ProMax, grouped by language. 70 repositories across 7 languages.
Language	Repository	License
C	betaflight/betaflight	GPL-3.0
	aviggiano/redis-roaring	MIT
	davidesantangelo/krep	BSD-2-Clause
	radareorg/radare2	LGPL-3.0
	CESNET/libyang	BSD-3-Clause
	arkq/bluez-alsa	MIT
	aws/s2n-tls	Apache-2.0
	bitcoin-core/secp256k1	MIT
	openssl/openssl	Apache-2.0
C++	deskflow/deskflow	GPL-2.0
	ETLCPP/etl	MIT
	nasa/fprime	Apache-2.0
	Icinga/icinga2	GPL-3.0
	LMMS/lmms	GPL-2.0
	OpenOrienteering/mapper	GPL-3.0
	WasmEdge/WasmEdge	Apache-2.0
	bloomberg/blazingmq	Apache-2.0
	biojppm/rapidyaml	MIT
Go	cli/cli	MIT
	go-gitea/gitea	MIT
	TecharoHQ/anubis	MIT
	restic/restic	BSD-2-Clause
	OpenListTeam/OpenList	AGPL-3.0
	caddyserver/caddy	Apache-2.0
	derailed/k9s	Apache-2.0
	gitleaks/gitleaks	MIT
	gohugoio/hugo	Apache-2.0
	grpc/grpc-go	Apache-2.0
	istio/istio	Apache-2.0
	jesseduffield/lazygit	MIT

Figure 5: Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.

Table 5: Per-language statistics for SWE-Bench ProMax, including number of repositories, instances, and average patch complexity.
Language	#Repos	#Inst.	Avg. #Files	Avg. LOC	Avg. #Non-test
C	9	20	17.9	424.1	15.2
C++	9	22	21.4	196.3	16.0
Go	16	23	16.0	227.4	9.4
Java	11	26	20.8	309.8	16.8
Python	18	29	10.6	299.8	7.0
Rust	5	22	14.5	284.8	11.0
TypeScript	2	28	11.9	122.6	7.5
Overall	70	170	15.9	261.6	11.4

Figure 6: Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring.

Table 6: Summary of representative instances in SWE-Bench ProMax, one per language, sorted by number of modified files.
Lang.	Repository	Files	LOC	Refactoring summary
C++	nasa/fprime	244	559	Unify header includes across framework
Java	plantuml/plantuml	94	1,629	Add hour-level time resolution to Gantt engine
C	betaflight/betaflight	62	846	Rename motor protocol configuration fields
Rust	tracel-ai/burn	49	1,084	Unify scalar arguments across tensor operations
Go	OpenListTeam/OpenList	47	608	Refactor upload stream buffering across drivers
Python	google/langextract	30	1,960	Centralize provider output-format handling
TS	ant-design/ant-design	27	97	Unify destroyOnHidden across components

Figure 7: Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.

研究结果

在六个前沿模型和两种智能体框架下评测,表现最好的GPT-5.2解决率仅为41.2%,远低于前沿智能体在SWE-bench Verified上超过75%的水平。
开源模型GLM-5和Qwen3.5在OpenHands框架下均达到36.5%,接近GPT-5.2(41.2%)和Claude Sonnet 4.6(38.8%),但每个实例成本仅0.24美元,约为Claude Sonnet 4.6(4.77美元)的二十分之一。
将智能体修改的文件数与黄金补丁对比发现,对于大规模补丁,智能体修改的文件数明显不足:黄金补丁的累积分布约在20个文件处达到90%,而智能体在约10个文件处就已达到90%。
失败的尝试消耗的交互轮数明显多于成功的尝试,这与智能体陷入反复编辑-撤销的低效循环有关。
不同语言间性能差异很大,没有单一模型在所有语言上都占优:Claude Sonnet 4.6在TypeScript(53.6%)和Rust(63.6%)上领先,GPT-5.2在Python(48.3%)和C(75.0%)上表现最好,GLM-5在Java(34.6%)上领先。

Figure 8: Co-occurrence matrix of task categories. API Interface Change and Refactoring Cleanup co-occur in 79 instances, while Bug Fix frequently accompanies both (33 and 48 instances respectively), reflecting how refactoring often surfaces latent defects that must be addressed concurrently.

可应用场景

可作为评测AI编程智能体处理大规模多文件重构任务能力的标准基准。
可用于比较不同智能体框架或提示策略对跨文件协同能力的影响。
为在开源模型与商业模型之间权衡解决率与成本效率提供参考依据。
可用于分析类型系统、内存模型等语言特性如何影响智能体在七种语言上的重构表现。

Figure 9: Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances.

局限与待验证事项

基准规模不算大(170个实例、70个仓库),部分语言的实例高度集中在极少数仓库中(例如TypeScript的28个实例仅来自2个仓库)。
评测仅使用了mini-swe-agent和OpenHands两种智能体框架,其他架构下的表现尚未验证。
任务类别和所需推理技能的标注(附录B)由Claude Sonnet 4.6完成,仅用于分析目的,不影响基准的标准答案或评测结果本身。
数据仅取自2025年1月之后的提交,这在一定程度上但并不能完全排除较新模型训练数据污染的可能性。

为什么重要

代码重构是实际软件开发中最常见的工作之一,要求在不改变外部行为的前提下协调修改多个文件,因此是检验智能体真实能力的更严苛、更贴近现实的测试。在现有基准趋于饱和、可信度受质疑的背景下,这一基准为衡量AI编程智能体的真实进步提供了一把尚未被'刷分'的新尺子。

本文术语

SWE-bench · 一个基于真实GitHub issue和代码仓库来评测AI编程智能体的知名基准系列
代码重构 · 在不改变代码外部可观察行为的前提下,对代码结构进行整理和调整
解决率 · 智能体修改后的代码通过全部测试用例的任务所占的百分比
黄金补丁 · 原始开发者实际提交的正确代码修改,作为参考答案
智能体框架 · 让模型能够循环执行查看文件、编辑代码、运行命令等操作的执行环境,如mini-swe-agent、OpenHands

论文原文摘要（英文）

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

作者 · Yuling Shi

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Yuling Shi et al., arXiv:2608.09802, CC BY 4.0