AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

arXiv:2608.098022026-08-09

When AI coding agents were tested on real-world multi-file refactoring, even the best model solved only 41.2% of tasks

Existing coding benchmarks are saturating, and an audit found nearly 60% of unsolved SWE-bench Verified instances have flawed tests. This paper introduces SWE-Bench ProMax, an expert-curated benchmark of 170 large-scale refactoring tasks drawn from real commits across seven programming languages. The best evaluated model reached only 41.2% resolve rate, showing that coordinated multi-file refactoring remains a genuine challenge for current AI agents.

METAL LAB explanatory visual

How SWE-Bench ProMax was built and evaluated

Evidence statusMeasured results reported

  1. Stage 1: Data collectionMined 29,782 candidate commits containing 'refactor' from GitHub repos with 500+ stars, open-source licenses, and 80%+ code in one of seven target languages, after January 2025
  2. Stage 2: Environment validationBuilt a Docker environment for each candidate commit and confirmed the gold patch actually passes the test suite, discarding failures
  3. Stage 3: Expert curationFiltered out low-complexity tasks, rewrote issue descriptions from scratch, and removed overly narrow or broad tests, yielding 170 final instances
  4. Model evaluationTested GPT-5.2, Claude Sonnet 4.6, Gemini-3-Pro, GLM-5, Kimi-K2.5, and Qwen3.5 under mini-swe-agent and OpenHands scaffolds; best resolve rate was 41.2%
  5. Failure pattern analysisFound agents modify fewer files than required on large patches and that failed attempts burn far more interaction rounds than successful ones
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. The work starts from the concern that existing benchmarks like SWE-bench Verified have become too easy and their test suites are unreliable, so scores no longer clearly reflect real capability.
  2. The team mined 29,782 candidate 'refactor' commits from GitHub repositories with at least 500 stars, approved open-source licenses, and at least 80% code in one of seven target languages, then filtered down to 170 instances through environment validation and expert rewriting.
  3. Every issue description was rewritten from scratch to remove ambiguity, and test suites were manually reviewed to remove overly narrow tests (which reject correct solutions) and overly broad tests (which check unstated requirements).
  4. The benchmark spans Python, Java, TypeScript, Go, C, C++, and Rust, with instances averaging 11.4 modified files and 261.6 lines of code per task, substantially larger than prior benchmarks.
  5. Six frontier models (Claude Sonnet 4.6, GPT-5.2, Gemini-3-Pro, GLM-5, Kimi-K2.5, Qwen3.5) were evaluated under two agent scaffolds, mini-swe-agent and OpenHands, with GPT-5.2 achieving the top resolve rate of 41.2%.
Figure 1: Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.
Figure 1: Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.
Table 1: Comparison of SWE-Bench ProMax with existing benchmarks. Avg. >5 Files: the average gold patch modifies more than 5 files. Expert Curated: problem descriptions and/or test suites undergo manual expert review or authoring beyond automated collection.
BenchmarkExecution BasedRepo LevelMulti LingualRefac- toringAvg. >5 FilesExpert Curated
HumanEval [5]✓✓
MBPP [2]✓✓
LiveCodeBench [19]✓
SWE-bench [21]✓✓
Multi-SWE-bench [49]✓✓✓✓
SWE-PolyBench [33]✓✓✓
SWE-bench Pro [8]✓✓✓
SWE-EVO [23]✓✓✓
Terminal-Bench [26]✓✓✓
RefactorBench [13]✓✓✓✓
SWE-Refactor [47]✓✓✓
SWE-Bench ProMax (Ours)✓✓✓✓✓✓
Figure 3: Data collection and curation pipeline for SWE-Bench ProMax.
Figure 3: Data collection and curation pipeline for SWE-Bench ProMax.
Table 3: Resolve rates (%) of all evaluated models across scaffolds on SWE-Bench ProMax.
OverallAvg. StepsAvg. CostPer-Language Resolve Rate
PyJavaTSGoCC++Rust
Mini-SWE-Agent
Proprietary
Gemini-3-Pro26.558.0$0.6017.215.414.326.150.045.527.3
Claude Sonnet 4.630.699.5$2.3213.819.232.113.065.040.940.9
GPT-5.221.825.2$0.1917.215.421.413.045.031.813.6
Open-weight
GLM-522.9108.9$0.1013.87.725.021.750.022.727.3
Kimi-K2.526.585.3$0.3717.223.121.417.460.031.822.7
Qwen3.520.6155.4$0.9317.27.710.713.045.027.331.8
OpenHands
Proprietary
Gemini-3-Pro19.451.2$1.4913.819.20.08.745.036.422.7
Claude Sonnet 4.638.8117.9$4.7717.230.853.626.150.036.463.6
GPT-5.241.2115.1$3.6048.319.235.726.175.036.454.5
Open-weight
GLM-536.5114.2$0.2420.734.628.634.865.045.536.4
Kimi-K2.532.999.6$0.7224.130.810.743.570.045.518.2
Qwen3.536.5141.2$0.7837.926.917.939.165.054.522.7
Figure 4: Language distribution in SWE-Bench ProMax.
Figure 4: Language distribution in SWE-Bench ProMax.
Table 4: Repositories and licenses used in SWE-Bench ProMax, grouped by language. 70 repositories across 7 languages.
LanguageRepositoryLicense
Cbetaflight/betaflightGPL-3.0
aviggiano/redis-roaringMIT
davidesantangelo/krepBSD-2-Clause
radareorg/radare2LGPL-3.0
CESNET/libyangBSD-3-Clause
arkq/bluez-alsaMIT
aws/s2n-tlsApache-2.0
bitcoin-core/secp256k1MIT
openssl/opensslApache-2.0
C++deskflow/deskflowGPL-2.0
ETLCPP/etlMIT
nasa/fprimeApache-2.0
Icinga/icinga2GPL-3.0
LMMS/lmmsGPL-2.0
OpenOrienteering/mapperGPL-3.0
WasmEdge/WasmEdgeApache-2.0
bloomberg/blazingmqApache-2.0
biojppm/rapidyamlMIT
Gocli/cliMIT
go-gitea/giteaMIT
TecharoHQ/anubisMIT
restic/resticBSD-2-Clause
OpenListTeam/OpenListAGPL-3.0
caddyserver/caddyApache-2.0
derailed/k9sApache-2.0
gitleaks/gitleaksMIT
gohugoio/hugoApache-2.0
grpc/grpc-goApache-2.0
istio/istioApache-2.0
jesseduffield/lazygitMIT
Figure 5: Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.
Figure 5: Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.
Table 5: Per-language statistics for SWE-Bench ProMax, including number of repositories, instances, and average patch complexity.
Language#Repos#Inst.Avg. #FilesAvg. LOCAvg. #Non-test
C92017.9424.115.2
C++92221.4196.316.0
Go162316.0227.49.4
Java112620.8309.816.8
Python182910.6299.87.0
Rust52214.5284.811.0
TypeScript22811.9122.67.5
Overall7017015.9261.611.4
Figure 6: Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring.
Figure 6: Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring.
Table 6: Summary of representative instances in SWE-Bench ProMax, one per language, sorted by number of modified files.
Lang.RepositoryFilesLOCRefactoring summary
C++nasa/fprime244559Unify header includes across framework
Javaplantuml/plantuml941,629Add hour-level time resolution to Gantt engine
Cbetaflight/betaflight62846Rename motor protocol configuration fields
Rusttracel-ai/burn491,084Unify scalar arguments across tensor operations
GoOpenListTeam/OpenList47608Refactor upload stream buffering across drivers
Pythongoogle/langextract301,960Centralize provider output-format handling
TSant-design/ant-design2797Unify destroyOnHidden across components
Figure 7: Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.
Figure 7: Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.

Findings

  • Across six frontier models and two agent scaffolds, the best model (GPT-5.2) achieved only 41.2% resolve rate, far below the 75%+ that frontier agents reach on SWE-bench Verified.
  • Open-weight models GLM-5 and Qwen3.5 both reached 36.5% under OpenHands, close to GPT-5.2 (41.2%) and Claude Sonnet 4.6 (38.8%), while costing only $0.24 per instance versus $4.77 for Claude Sonnet 4.6, roughly one-twentieth as much.
  • Comparing files modified by agents against the gold patch showed agents modify far fewer files for large patches specifically: the gold patch CDF reaches 90% around 20 files, while agents reach 90% by about 10 files.
  • Failed attempts consumed substantially more interaction rounds than successful ones, consistent with agents falling into unproductive edit-revert cycles.
  • Performance varied widely by language with no single model dominating: Claude Sonnet 4.6 led on TypeScript (53.6%) and Rust (63.6%), GPT-5.2 led on Python (48.3%) and C (75.0%), and GLM-5 led on Java (34.6%).
Figure 8: Co-occurrence matrix of task categories. API Interface Change and Refactoring Cleanup co-occur in 79 instances, while Bug Fix frequently accompanies both (33 and 48 instances respectively), reflecting how refactoring often surfaces latent defects that must be addressed concurrently.
Figure 8: Co-occurrence matrix of task categories. API Interface Change and Refactoring Cleanup co-occur in 79 instances, while Bug Fix frequently accompanies both (33 and 48 instances respectively), reflecting how refactoring often surfaces latent defects that must be addressed concurrently.

Where it can be used

  • Can be used as a standard benchmark to evaluate how well AI coding agents handle large-scale, multi-file refactoring tasks.
  • Useful for comparing how different agent scaffolds or prompting strategies affect cross-file coordination ability.
  • Provides a reference point for weighing resolve rate against cost efficiency when choosing between open-weight and proprietary models.
  • Enables analysis of how language-specific features (type systems, memory models, etc.) affect agent performance across seven languages.
Figure 9: Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances.
Figure 9: Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances.

Limits and open work

  • The benchmark is relatively small (170 instances, 70 repositories), and coverage for some languages is concentrated in very few repositories (e.g., TypeScript's 28 instances come from just 2 repositories).
  • Evaluation was limited to two agent scaffolds, mini-swe-agent and OpenHands, so results with other agent architectures are untested.
  • Task category and reasoning-skill labels (Appendix B) were generated by Claude Sonnet 4.6 for analysis purposes only and do not affect the benchmark's ground truth or evaluation outcomes.
  • Commits were restricted to those made after January 2025, which limits but does not fully guarantee against training data contamination for newer models.

Why it matters

Refactoring is one of the most common real-world software engineering activities, requiring coordinated, behavior-preserving edits across many files, which makes it a more realistic and demanding test of agent capability. As older benchmarks saturate and lose credibility, this benchmark gives the field an unsaturated, quality-checked yardstick for measuring genuine progress in AI coding agents.

Terms in this paper

  • SWE-bench · A well-known family of benchmarks that evaluates AI coding agents using real GitHub issues and repositories
  • code refactoring · Restructuring code without changing its observable behavior
  • resolve rate · The percentage of tasks where the agent's code changes pass all tests in the suite
  • gold patch · The original developer's actual code changes, used as the reference solution
  • agent scaffold · The execution framework (e.g., mini-swe-agent, OpenHands) that lets a model view files, edit code, and run commands in a loop

Original abstract (English)

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

Authors · Yuling Shi

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Yuling Shi et al., arXiv:2608.09802, CC BY 4.0