OpenAI Carried 70 Million Requests a Second on Python

The part worth watching is not the size but the speed. The company wrote that engineers usually build for 10x and hope the design holds for a few years, while Habitat has grown more than tenfold year over year for three years running. That meant a continuous run of decisions to squeeze the last drop out of whatever was already in hand, buying time for the foundations. Jon Lee, Chaomin Yu and Ben Ries, all members of technical staff, laid out that list of decisions in order.

The first decision was to pull the library out into a service. In mid-2024 Habitat was a small Python library sitting next to ChatGPT's main server, and all it did was spare product engineers from thinking about schema lookup, routing, authorization and encryption. By the middle of 2025 the client-side approach had hit its ceiling. The authors wrote that "changes to the client library necessitated complex coordination across dozens of services, a process that proved increasingly brittle, inefficient, and susceptible to operational failures."

An actual incident pushed that conclusion through. The job was to split critical data across Azure Cosmos DB accounts in several regions so a single region going down would not spread the damage, and simply putting the routing logic into the client behind a feature flag and rolling it out to every service took days. Adding a shadow run to check the logic took several more days, and fixing what turned out to be wrong took several more. Right before the flag went on, one team rolled back to an older client for unrelated reasons, and the very outage they had been working to avoid happened anyway.

The second decision was not to abandon Python right away. The company went ahead knowing that running the service in Python would add network latency and inflate CPU and memory costs. The authors explained that "our primary objective then was not cost or resource optimization, but rather unblocking product developers and achieving platform stability." They pinned the choice down as a deliberate incursion of technical debt, and wrote that it would not hold at 100x scale, making an eventual rewrite close to certain.

While Python held, the thing this team wrestled with longest was tail latency. The authors wrote that "when the average user request results in hundreds of database calls, the slowest database call is the one the user feels." Python's asyncio overlaps I/O but does not share the CPU, so once compression, encryption and checksumming pile up, a response can already be sitting there with no turn available to read it. The company said this scheduling delay stretched to hundreds of milliseconds and, in bad cases, several seconds. So they kept the number of concurrent requests per process very small and massively scaled out the number of processes instead.

The way they found the cause is specific. When the service first launched, CPU profiling identified the culprit as the feature flag configuration file. By default the flag tool re-read a large config containing every production rule for every service once a minute, at exactly the same moment each time, with no jitter. With as many as eight Python processes per pod, every minute all the workers stopped what they were doing at once and parsed that large file. The fix was to ship a smaller targeted config, lengthen the refresh interval and scatter the timing.

Changing the order in which connections get reused is the more interesting part. One day they turned off a client that had been overloading the service, and yet a subset of processes kept getting worse, drawing more requests until they were restarted. The cause was that Python's aiohttp connection manager reuses the most recently returned connection first. The slower a server was, the later it returned its connection, so that connection was picked next, creating a feedback loop that stacked more work onto the struggling processes. Switching to reuse in the order connections were returned broke the loop and even reduced the usual variance.

Some things were deliberately left out of the design. Habitat does not let clients compose arbitrary SQL and instead exposes only a simple NoSQL API. The authors wrote that "the lack of a powerful API is an explicit tradeoff in Habitat's design." Back in the Postgres era, the company admitted, it was common for one expensive query on a hot path to take the whole database down, and now expensive queries are made conspicuous on the client side.

The limit the company wrote down for itself is the graph. An object and the edges attached to it are kept in the same storage partition, but no effort is made to keep the objects those edges point to nearby as well. Scaling out horizontally gets easier, but every hop across an edge may require pulling data from a different account in a different region. Teams that need complex queries get a separate analytical copy built by streaming change records, and the company admitted this arrangement puts extra friction on the teams using it.

Then, in the second quarter of this year, Python was retired. Two engineers rewrote the entire service in Rust with Codex and GPT-5.5, Rust now takes 95% of production requests, and the company says Python will be shut off entirely within weeks. By the company's own numbers Rust is 6x more CPU efficient and 15x more memory efficient, with markedly lower average and tail latencies. Python's peak was more than 20 million requests per second. METAL has reported that OpenAI opened the harness that runs Codex through its API, and this post is the record of the company using that tool first on its own infrastructure.

The sentence that stands out most in the announcement METAL read through is the calculation behind that decision. The authors wrote that "we bet that by the time a full migration off Python was required, Codex and GPT would make that migration achievable. That bet eventually proved correct." They took on technical debt on purpose while folding into the equation the fact that they were building the very tool that would pay it back. Habitat is now the second-largest service at OpenAI by core count, and fourth by Envoy footprint.

The lesson this post leaves is not about choosing a language. The company wrote that designing around simple, predictable, constant-work requests makes systems substantially easier to scale and harder to get wrong or misuse. The reason Habitat could ride Python past 20 million requests per second was the narrow API it had drawn around itself. The strength to absorb scale comes not from better materials but from a list, decided in advance, of what the system will not do.