Specific Releases Real-SWE Enterprise Code Benchmark

What separates Real-SWE from existing benchmarks is where the problems come from. Every task was taken from a private production codebase at a real company and reflects work its engineers actually handled, the team writes. Getting tax right on an invoice, migrating customers onto another identity system, metering usage: changes that shake the business if they go wrong. According to Specific, the codebases were selected from companies including an events service with more than 200,000 users and a top 100 App Store ranking, and a consumer fintech platform processing more than 100,000 bank statements.

The measurement design does not isolate the model either. To reflect how enterprise engineers actually work, models were scored together with their harnesses: Fable 5.1 and GLM 5.3 in Claude Code, GPT-6 Astra and GPT-5.6 Sol in Codex CLI, Gemini 3.8 Flash in Gemini CLI, Grok 4.6 in Grok Build, Muse Spark 1.3 in Muse Code, and Kimi K3 in Kimi Code. Ten tasks were run eight times per model for 640 rollouts in total, and the resolution rate is pass@1 averaged over those eight runs. Agents ran in isolated sandboxes and verifiers were injected at grading time, drawn from or lifted verbatim out of tests that already existed in the codebase.

The ranking runs Fable 5.1 at 38.8%, GPT-6 Astra at 33.8%, Gemini 3.8 Flash at 31.2% and GLM 5.3 at 28.8%, followed by Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 at 18.8% and GPT-5.6 Sol at 16.2%. Six of the ten tasks landed below a 15% resolution rate. The analytics stream reducer task went 64 attempts across eight configurations without a single pass, the tax jurisdiction task stopped at 3.1% and the linearizable scan at 4.7%. At the other end, the multi-region sweep reached 67.2% and API keys and environments 65.6%.

More interesting than the scores is the shape of the mistakes. Grouped by failure type, the most common was leaving out behavior the instruction required. Grok 4.6 hit that in 41 of its 61 failed runs, or 67.2%, and Kimi K3 in 35 of 65, or 53.8%. For GPT-5.6 Sol, 43.3% of failures came from building on a guess about the system instead of checking it in the workspace, while 49.1% of Gemini 3.8 Flash's failures were integration errors, the right idea wired into the surrounding system incorrectly. Regressions that broke existing behavior stayed under 10% for most configurations, and delivering the change to a file the running application never calls peaked with GLM 5.3 at 7.0%.

Spending more time did not help either. Rollouts that finished within ten minutes failed in 70 of 98 runs, or 71.4%, and rollouts that ran ten minutes or longer failed in 398 of 542, or 73.4%. Staying on the problem longer did not mean getting closer to the answer. Cost did not buy a score either. The estimated cost of a single rollout ranges from $2.50 to $6.96; Fable 5.1 was both the most expensive at $6.96 and the top scorer, but the cheapest, Gemini 3.8 Flash at $2.50, delivered 31.2%, while GPT-5.6 Sol at $2.65 finished last.

The report also explains why the team went to the trouble of borrowing other companies' code. Tasks on private codebases are natively out of distribution, meaning no model is likely to have seen them in training. "99% of tokens in real-world enterprises are hidden away from the frontier models," the report states. The instructions are not generous either. The median Real-SWE instruction runs 1,742 characters and the reference solution edits a median of 11 files, against 6 files for the FrontierCode and DeepSWE comparisons cited. Agents get a short description and have to discover the implementation details themselves. The team also notes on its own that the instructions are somewhat underspecified. The report says the task prompts are "slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions." The companies that lent their codebases are not named, and viewing the sample tasks used in the analysis requires requesting access.

Opening up one of the tax tasks shows the difficulty. In a billing service where every business settles tax differently, some maintain their own rate, some want each invoice priced against the buyer's destination by an external tax authority provider, and some collect nothing at all, while a customer registered as exempt must not be taxed regardless of how its business is configured. The sandbox and production environments of the tax provider have to be split according to the account, an address the authority refuses must be reported without halting the invoice, and once a sale is settled it has to be filed back under that invoice's number so the returns reconcile. The environment is a NestJS service written in TypeScript wired to the tax provider's sandbox and production endpoints and a time-series ledger, and work a person would solve by asking the colleague next to them the agent has to infer from code and tools alone.

This scorecard reads clearly when set beside recent coding-model news. METAL has reported that Cognition drew a cost-performance frontier between American and Chinese models on its own benchmark, and has reported that Devin gained the ability to test and prove its own work with GPT-6 Astra. While scores on public benchmarks climb quickly, the same models produce a very different scorecard when the work carries money inside somebody else's company.

In the report METAL read, the team writes that company-specific engineering patterns matter, adding that "today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions." Asked whether AI-written code meets the bar of a real enterprise, the answer the team gives is that we are far from that reality. The bottleneck in bringing coding agents into a company is not the model's intelligence but the context the company has accumulated over years, and that context does not shrink on its own as benchmark scores rise. The way to shrink it is for the company to write down its own rules and verifications first, in a form an agent can read.