A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
arXiv:2608.183892026-08-20
Table 1: Catalog of semantics-preserving transformations (SPTs).| Transformation | Summary |
|---|
| If Else Switcher | Swaps if/else branches and negates the condition |
| For Loop Rewriting | Rewrites a for loop using an explicit iterator |
| And Condition Splitter | Decomposes if A and B into nested ifs |
| Comparison Swapper | Swaps operands and inverts the operator |
| While Loop Unrolling | Unrolls one iteration of a while loop |
| Double Negation Injector | Wraps a condition in not not (⋅) |
| Commutative Operand Permuter | Reorders commutative operands |
| Local Variable Renamer | Renames safe local variables to synonyms |
| If True Wrapper | Wraps a block in a permanently true guard |
| Try Except Injector | Wraps a block in a redundant try/except |
| Dead Code Injector | Inserts an unreachable block |
| Dead String Assignment | Inserts an unread variable assignment |
| Dead Method Injection | Appends an unreachable method to a class |
| String Literal Splitter | Splits a string literal into a concatenation |
Table 2: Comparison of mini-SWE and OpenCode Agent Scaffolds| Category | mini-SWE | OpenCode |
|---|
| Architecture | Single, centralized agent | Primary agents invoke tools and subagents for specific tasks (e.g., Build, Plan, General, Explore) |
| Available Tools | bash | bash, edit, write, read, grep, glob, lsp, apply_patch, skill, todo_write, web_fetch, web_search, question |
| Access Restrictions | None | Available tools determined by an agent’s role and user configuration |
| Context Management | Continuous, append-only | Multiple, dynamically changing contexts for different subagents |
无法转载的图表
- Figure 1: Mean degradation Δ¯ (percentage points) for each model under each scaffold on both benchmarks. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 2: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right), restricted to runs the agent resolved in both cases. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 3: Mean resolve rate (%) on unperturbed and perturbed runs for each model under each scaffold, on SWE-bench Verified (top) and SWE-bench Pro (bottom).
- Figure 4: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right). Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 5: Per-instance degradation Δ(i) for the 28 SWE-bench Verified instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
- Figure 6: Per-instance degradation Δ(i) for the 26 SWE-bench Pro instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
在原文中查看图表 →AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rew
作者 · Hasan Najib Mahmud (Colorado State University), Shreya Gupta (Microsoft), Isha Chaudhary (University of Illinois Urbana-
在 arXiv 阅读
