To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

What they did

1. The authors compared patches from five leaderboard models (GLM-4.6, GPT-5, Kimi-K2, Opus-4.5, Salesforce SAGE) on official SWE-bench Verified submissions against the human developer patch, finding mean deletion recall (the share of developer deletions the model also performs) of only 65.2% to 71.7% on 197 tasks all five models solved.
2. Models locate the correct file for over 92% of required deletions but remove the exact line in only 44.6% to 51.6% of cases, showing the failure is not localization but a reluctance to actually delete found code.
3. An LLM classifier found that 29.0% of passing patches follow a 'Guard-and-Go' pattern: the model keeps the developer-removed logic and wraps it in a new conditional or fallback path instead of deleting it.
4. Retrofitting 34 tasks with tests that specifically check whether the target code was deleted caused four frontier models' pass rate to drop from 63.2% to 41.9%, showing standard tests rarely catch missing deletions.
5. On CanItDelete, a new 200-task benchmark where the entire required edit is deletion, even the best model (Claude Opus 4.8) succeeded on only 79.0% of tasks, and even when given the exact lines to delete, GPT-5.6 Sol still failed 19.5% of tasks by deleting too much or adding code instead. Adding 12.8k deletion examples (0.7% of training tokens) to a 7B model's post-training mixture improved CanItDelete success by 13.9 points, and also raised SWE-bench Verified by 5.3 points and CanItEdit by 1.4.

Findings

• Across 197 tasks solved by all five leading models, mean deletion recall ranged from 65.2% (Kimi-K2) to 71.7% (Opus-4.5), and it fell further to 19.8-30.4% on the 57 tasks all five models failed.
• 29.0% of passing patches (494 of 1,703) followed the Guard-and-Go strategy, which passed SWE-bench Verified 72.2% of the time — lower than Delete-and-Replace (85.2%) but higher than non-reference alternatives (39.6%).
• Adding deletion-sensitive checks to 34 tasks dropped the pass rate of four frontier models from 63.2% to 41.9% (a 21.3 percentage-point decline), and 29 of 86 originally passing attempts retained the target code that should have been removed.
• On the 200-task CanItDelete benchmark, success across 12 models ranged from 79.0% (Claude Opus 4.8, best) down to 18.0% (weaker open models); even when given exact deletion spans, Claude Opus 4.8 reached 97.7% but Qwen3-235B still left target code in 17.5% of tasks.
• Adding 12,821 deletion examples (0.7% of training tokens) to a 7B model's post-training mixture raised CanItDelete success from 6.5% to 13.7%, improved SWE-bench Verified by 5.3 points and CanItEdit by 1.4 points, with EditBench essentially unchanged.

Where it can be used

• Adding a review checklist item that checks whether code the developer patch was meant to remove has actually been deleted, separate from whether tests pass, before merging AI-generated pull requests
• Adding deletion-sensitive tests (checks that fail if a target still remains, used as FAIL_TO_PASS criteria) into automated patch evaluation pipelines to catch Guard-and-Go style patches
• Designing post-training data mixtures for code-editing models that include a small share of deletion-specific examples to reduce deletion avoidance
• Using deletion-focused checks before deploying LLMs for legacy code cleanup, refactoring, or dead-code removal tasks

Limits and open work

• The analysis covers five specific models' latest 2025-era SWE-bench Verified submissions under a fixed OpenHands scaffold, so findings may not generalize to other scaffolds or later model generations
• The deletion-sensitive check treats the developer's removed code as the required target, but an alternative repair that preserves that code could still satisfy the original behavioral specification, so deletion is not proven to be the only valid fix
• The 34 deletion-sensitive-check tasks and the 200 CanItDelete tasks were both deliberately selected for heavy or exclusive deletion content, so they do not represent SWE-bench Verified or general repository work as a whole
• The deletion-focused post-training experiment is a proof of concept run on a single 7B in-house model with one training recipe, and whether the effect holds at larger scale is left to future work
• Adding deletion training reduced incomplete deletion but increased over-deletion (removing beyond the intended boundary), showing that completing a deletion and controlling its boundary are separate objectives that still need to be addressed

Why it matters

This provides concrete evidence that AI-generated code patches can pass all tests while still leaving obsolete code behind, making codebases harder to maintain over time. Anyone reviewing or deploying AI coding agents should not treat 'tests pass' as proof of a clean edit, and may need to add deletion-specific checks.

Terms in this paper

• Deletion Avoidance · A systematic tendency for a model to retain code that an intended edit actually requires removing
• Guard-and-Go · A patch pattern where the model keeps the code that should be deleted and wraps it in a new condition or bypass instead
• Deletion Recall · The share of the developer's actual deletions that the model's patch also performs
• CanItDelete · A new benchmark of 200 tasks mined from real commits where the entire required edit is deletion
• Deletion-Sensitive Check · A test added specifically to fail if the code that should have been deleted still remains

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3