Devin Has Started Proving Its Own Work

OpenAI posted the Cognition case study on its own site on September 11. Cognition builds Devin, an autonomous software engineer used by businesses around the world, from big banks to tech-native startups. The write-up says that as Cognition's engineering teams write more code, reviewing that work has become a challenge. What the company saw in GPT-6 Astra was not a smarter author but the ability to test its own output and show the result.

Walden Yan, co-founder of Cognition, put it in a single sentence. "One of the big pieces that Astra improves on is its ability to test and prove that its work actually functions the way you expect," he said. The distance between claiming something works and showing that it works is what this case study is about.

There is one concrete scene. According to the write-up, Devin uses Astra to test Otter Run, an iPhone game, and returns a recording of the game running in a simulator. Attached to the recording is a report listing the checks that passed and the areas left untested. The recording shows how the software actually behaves; the report records how far the testing reached.

The limitation the company acknowledges should not be skimmed past. A dedicated field for untested areas means the product openly carries the fact that this setup does not verify everything. Engineers read that field to find what still needs a human. Automation here does not remove review; it narrows where review should go.

The same method reached customer support. According to Yan, when a customer sends a screenshot of a bug, the team passes the image to Devin using Astra, which fixes the issue and returns a screenshot showing the result. He said this lets the team get back to customers much quicker. A bug report now arrives as an image and leaves as one.

The scope is not limited to a single product. "We've been using it to make improvements across our product, including the core cloud agent that is Devin, but also our CLI and desktop products," Yan said. The company is building what it sells with what it sells, and that kind of self-application is where a tool's weaknesses surface first.

A second case study published the same day widens the picture. OpenAI also posted one on Perplexity, which builds an answer engine. Johnny Ho, cofounder and chief strategy officer of Perplexity, said, "We can have the model craft communications, edit real-world systems, and monitor our production software in a way that previous generations were not able to." The phrasing separates the ability to write from the ability to touch running systems.

Perplexity's example is testing too. Ho explained that he uses Astra to generate test programs that convincingly imitate the responses of outside services such as language model APIs or connectors, validating end-to-end workflows without checking each step by hand. Then he described the level of trust this way: "We're actually able to trust it with full end-to-end systems and check in on it much less frequently than previous generations of models."

What matters is that both cases are saying the same thing. The signal that agents have improved is showing up not as a benchmark score but as how often a person has to look. Yan described what he expects next: "We expect over time that we have to manually look at less code and end up shipping more at the end of the day," he said. The target is plainly review cost, not code quality.

Cognition is not only a consumer of other companies' models. METAL reported that it released its own coding model, SWE-2, and placed it on the frontier between performance and cost. A company with its own model is using a rival's model as the tester inside its product, a sign that model choice is splitting by task rather than by vendor.

The two case pages METAL read contain almost no numbers. Not one line claims a speedup of so many times or a defect reduction of so many percent. Instead they list only artifacts a person can look at: recordings, reports, screenshots. Companies are describing a change that has not yet hardened into a benchmark by handing over an inventory of evidence instead.

From an engineering standpoint, what changed is the output format of the agent. Where a block of code once came out for a human to read and judge, code now arrives together with footage of that code running and a document stating what was checked and what was not. The reviewer's job shifts from decoding code to examining evidence. The open question is who inspects that evidence, and right now the same model is sitting in that chair.

A system that tests the code it wrote has a built-in hole. From the outside it is hard to tell fitting the code to the test apart from fitting the test to the code. That is why the untested-areas field is the most important item in this product. As automation grows, what gains value is not the output but the marker of limits attached to it, and the answer Cognition and Perplexity offer together is not removing people from review but shrinking what they have to look at.