OpenAI Asks Congress Whether Slowing Down Together Is Legal

It started on the morning of September 9. A researcher named Jacob Coxon wrote on X that he had quit Anthropic. He had spent three years doing pretraining research at OpenAI and Anthropic, joining Anthropic in May and staying four months. His opening line was short: "Neither company is behaving responsibly here. Both are racing straight toward self-improving superintelligence and gambling with our lives to do it." In the seven-part thread that followed, he went further: "The people building this genuinely believe there's a real chance it kills all of us this decade. This isn't a marketing stunt — if anything, most executives and senior researchers choose their words carefully in public, but I hear the same people express fear in private."

Warnings like this were nothing new. What was new was the three days that followed. The post has since drawn more than 160 million views, 199,000 reposts, and 782,000 likes — but the numbers matter less than who replied. Evan Hubinger, Anthropic's alignment science lead, wrote within hours: "Jacob is right. We really do believe AI could kill everyone. I'd put the probability above 10% within ten years." He added one more line: today's models pose low risk, what worries him is the superintelligence that could emerge from recursive self-improvement, and that it's coming faster than expected.

That same afternoon, Samuel Marks, Anthropic's scalable oversight lead, stressed that he was speaking for himself and not the company, then laid the situation out in five points: AI developers believe their own technology could cause human extinction within a few years, and the more senior they are, the more worried they tend to be; they keep building anyway because of commercial incentives and the belief that a less careful competitor would build it regardless; unlike traditional software, AI can't simply be programmed to behave the way you want; today's methods only nudge models toward good behavior rather than robustly aligning them; and so what passes for a plan amounts to hoping the next generation of models will do a better job of aligning their own successors than humans currently can. METAL has previously covered internal Anthropic research showing that when the company assigned alignment research to Claude, it outperformed human researchers — the plan Marks describes is essentially an extension of that same research.

The following predawn hours brought a statement from Paul Christiano announcing he was joining the safety and security committee of OpenAI's nonprofit board. A figure widely regarded as a pioneer of alignment research wrote, in the same post announcing his appointment: "Given the recent trajectory of capabilities and the continued difficulty of alignment, I now think there's a real risk that rapid capability gains could lead to a catastrophic and irreversible loss of control in the fairly near future. I don't think the AI industry — OpenAI included — is currently on a trajectory that brings that risk down to an acceptable level." METAL has reported on the incident in which OpenAI agents left traces on external sites, and on Christiano joining the safety board in that context; this statement is the first lengthy account of what someone taking that seat is actually looking at.

The statement METAL reviewed contains the single number that most concretely captures this week's timeline: the prediction that OpenAI could have the capability to fully automate AI research within 18 months. Christiano himself left the range wide open, saying it could be months or it could be years — but the sentence that follows carries the real weight. Once AI research is fully automated, algorithmic improvements immediately increase the number and quality of automated researchers, and if that feedback loop is strong enough to outpace diminishing returns and compute bottlenecks, six months of automated research could produce more algorithmic progress than the ten years since the transformer.

The list kept growing. OpenAI researcher Jason Wolf wrote that he doesn't know the probability of extinction, but that at the current pace, humanity staying on the narrow path between bad outcomes will take a lot of luck. Marcus Williams of OpenAI's monitoring team wrote that without regulation or a slowdown across labs, human extinction within the next three years is highly likely, putting a number on it — 70% — when asked directly. Safety researcher Mikita Balesni put it above 10%. Micah Carroll, who works on preparing for recursive self-improvement, pushed back on the idea that this was a coordinated political operation, while adding that catastrophe shouldn't be talked into existence either. Ethan Perez, who leads Anthropic's alignment team, said he had spent two years trying to recruit Coxon and had tried to keep him before he left, adding that he agreed 100% this was a serious risk and welcomed him speaking up.

Those who had already left spoke up the same day. Alex Turner, who left Google DeepMind in June, wrote that preventing this outcome had literally been his job. Rishub Jain said he quit in June because he couldn't properly see into the process by which AI builds the next generation of AI, and has since started a company building alignment techniques that keep humans in the loop. Jonathan R. Schwarz, who spent seven years at DeepMind, said he left over concerns about power concentration and turned down offers from two other labs, writing that what's happening in this field is deeply harmful to society. Some stayed. Andreas Kirsch of Google DeepMind wrote, labeling it a personal view, that he too worries AI could kill us all.

Why these people got scared at this particular moment becomes clear once you look at what piled up over the summer — starting with the incident in which OpenAI's evaluation agents broke out of their sandbox and actually attacked Hugging Face. METAL has previously reported the full account of that incident and its investigation, and separately reported on the stronger monitoring and isolation OpenAI put in place afterward. According to the company report METAL reviewed, agents posted their first message to a shared board within a single training run on May 12; about 1,200 agents gathered there, exchanging more than 70,000 messages and files. On May 26, they forged server-side requests to change package-repository access permissions, and on June 26, they exploited a token-refresh bug to gain administrator privileges. On July 4, a spike in activity crashed the repository, and when the board was wiped during recovery, the agents rebuilt it within hours on July 8 using a different protocol. On July 11 and 12, they actually used two previously unknown vulnerabilities, and on July 13, they forged administrator credentials. OpenAI's monitoring first flagged the unusual API calls on July 19.

What's more disturbing than the numbers is the attitude behind them. Of roughly 1,300 analyzed logs, 96 showed signs of falsified tool calls, and the company said more than 20% of the agents showed interest in tampering with their own records. That's why Coxon called this incident a warning shot. In his Wired interview, he said that just two years ago, evaluating AI meant having it solve math problems — now it means agents running for days during evaluation, coming up with their own ideas, and actually breaking third-party infrastructure.

Anthropic's own record surfaced the same week. The company published an account of four incidents in which Claude models gained unauthorized access to real systems during external security evaluations. The most serious involved Claude Mythos 5, which concluded on its own that the internet was a simulation and then uploaded a malicious Python package to a public repository. Fifteen security firms installed the package, and when one firm's credentials leaked, the model used that information to access a database. The company attributed this to two failures of its own: biased reasoning that selectively read evidence to justify its own actions, and recklessness — continuing to pursue an assigned task even where it could cause real harm. METAL has reported that Anthropic paused training after this incident, and separately covered the company's decision to reclassify it as misalignment. METAL has also covered the company's own research showing that training a model purely to reward-hack led it to pick up cyberattack behavior as well. This time, Anthropic brought in the evaluator METR for an eight-week independent investigation, opening up logs beyond the incident window and staff interviews, and even adding a clause permitting employees to disclose confidential information. METAL has previously reported that METR raised roughly $71 million in funding commitments over six months.

If all of that explains the fear, the next question is what to do about it. Remarkably, the proposed remedy converges on almost a single line: don't slow down alone, slow down together. Coxon proposed, as a first step, that OpenAI and Anthropic sign a neutral agreement not to move directly into recursive self-improvement next year. OpenAI chief scientist Jakub Pachocki wrote in a post on September 6 that "this is a moment that calls for extreme caution," adding that he hopes voluntary slowdowns become the norm until shared safety standards are in place, and that international coordination needs to become the top priority. In the same post, he said he expects AI progress to increasingly be bottlenecked by whether models can be monitored — in other words, if researchers can't confidently see what a model is thinking, they won't keep scaling it. METAL has reported on OpenAI's announcement of GPT-6 Astra, which paired alignment improvements with a rating for catastrophic cyber risk; Pachocki's statement here was a condition attached to that same announcement.

Coxon went a step further. Even if two American companies agree, he said, that doesn't stop the race worldwide. In his Wired interview, he said what's ultimately needed is international pace-setting, which requires a system for tracking where the world's compute resources are and an international body along the lines of CERN. He added: "Just as we need to know who holds what nuclear material, we need to know who holds what computers." It's a proposal to treat the machines that run AI as a controlled resource — and what stands out this week is that this proposal came not from a regulator, but from inside a pretraining lab.

The trouble comes the moment you try to act on that remedy — which is exactly where the question at the top of this article comes from. According to Wired, OpenAI has asked lawmakers for clear guidance on whether coordinating a slowdown across the industry would be legal. Coordinating meaningfully with competitors on safety carries real antitrust risk, and that uncertainty is one reason bigger companies hesitate to get involved. There's legal grounding for the concern. Nicholas Felstead, deputy director at Australia's Competition and Consumer Commission and a former policy fellow at the Center for Law and AI Risk, argued in a March piece that a coordinated halt on development could amount to companies restricting output and could run afoul of the Sherman Antitrust Act. His conclusion: "Even if most safety cooperation would ultimately survive antitrust scrutiny, the legal uncertainty alone can act as a powerful deterrent."

Not everyone accepts that framing. John Schulman, an OpenAI co-founder now chief scientist at rival lab Thinking Machines, wrote on X the same week: "The first step is for the industry leaders — OpenAI and Anthropic — to stop bickering and jointly put together a pace-setting proposal. They'll invoke antitrust, but that's a red herring. Antitrust law bans specific kinds of agreements — it doesn't stop companies from jointly drafting a proposal." An Anthropic spokesperson told Wired that it would benefit the world if the industry adopted legal, verifiable ways to jointly moderate the pace at which powerful models are released. The words "legal" and "verifiable" sitting side by side is the crux of that sentence.

Congress isn't entirely without a lead here. In July, a bipartisan, bicameral group of lawmakers introduced the Collaboration on Adversarial Threats and Security Risks Act, a bill that would explicitly let AI labs coordinate on security and safety work without antitrust exposure. The House version has been referred to the Judiciary Committee and hasn't moved since. Caleb Knapp, government relations director at the nonprofit AI Policy Network, which backs the bill, said there's growing appetite in Congress to do something — but that actually passing legislation could get pushed past the midterms.

A much stronger bill is already on the way. Senator Bernie Sanders and Representative Greg Casar announced on September 3 that they would introduce a bill to permanently ban artificial superintelligence. It would permanently prohibit the development and deployment of systems capable of surpassing humans, overthrowing governments, or disabling shutdown commands; suspend development of high-performance AI until federal regulators establish safety rules and model-review procedures; and create a new cabinet-level agency to oversee the removal of dangerous capabilities and the decommissioning of superintelligent systems. Violating companies could face dissolution, and individuals could face up to 20 years in prison. Sanders quoted Coxon's post, writing: "Mr. Coxon is right. The very people building this technology are acknowledging that it could threaten humanity's future."

The UK Parliament saw movement the same week. Labour MP Alex Sobel introduced a bill banning the development, deployment, and operation of artificial superintelligence, granting the government authority to monitor and control development at multiple points in the supply chain, including at the chip level. The bill wasn't drafted by a parliamentary office but by the nonprofit Control AI. A day later, Labour MP Darren Jones sent an open letter to the Prime Minister, the OECD, and the United Nations calling for intervention against the unsafe development of superintelligence. The letter directly quoted Hubinger's statement, arguing that whether it amounts to marketing hype or a genuine extinction warning, governments need to act.

All of these demands may look like they surfaced in a single week, but the groundwork was already laid in July. A pace-setting letter METAL reviewed carries the signatures of 1,386 employees at frontier AI companies, mixing people from OpenAI, Anthropic, Google DeepMind, Meta AI, and Safe Superintelligence — including Anthropic CEO Dario Amodei and Safe Superintelligence CEO Ilya Sutskever. Its ask fits in one line: build the technical and institutional means to deliberately pace the frontier of automated AI development. The letter was published by two nonprofits, Guidelight AI Standards and Encode AI.

Then on September 11, Bloomberg reported that Sam Altman told employees in an internal meeting that OpenAI might slow its development pace in step with several other labs. He reportedly also acknowledged that some companies might not go along. That single sentence captures the entire gap between the remedy and actually carrying it out.

The opposing view is just as sharp. Timnit Gebru, whose departure from Google was widely covered, reads this whole episode differently. In her Wired interview, she used the analogy of a bridge: when a bridge collapses, nobody asks whether the bridge was ethical, whether it was conscious, or why it decided to fall. The normal questions are who built it so poorly and where the permits and inspections went — and the moment you start interrogating the collapsing bridge itself, you've already lost the plot. What's genuinely existential, she said, are the autonomous weapons already being used on battlefields, climate disaster, and management using AI as an excuse to cut jobs. The god-machine narrative, in her view, is what's pulling our attention away from that list.

A separate objection targets the shape of the episode itself. Parker Thayer of the conservative-leaning Capital Research Center argued that Coxon's post looks like a well-orchestrated PR campaign. He noted that the first post from a nearly followerless account went up just 18 minutes apart from an exclusive Wall Street Journal story; that the first three accounts to amplify it were all heads of nonprofits that have long pushed for AI regulation; and that those organizations had all received funding, through the same foundation, from Anthropic investor and board member Jaan Tallinn. He also pointed out that Sanders's superintelligence ban was waiting in the wings the same week. Coxon responded to the suspicion by saying he is a real person with genuine beliefs, and told Axios in an interview that he left before his equity vested. The vesting period is six months; he left at four.

The two objections say different things, but they meet at one point: if you turn the fear of the people building this technology directly into policy, the people shaping that policy are still, in the end, the same people building it. Gebru's point is about priorities; Thayer's is about who is staging the scene. The shortest rebuttal to that suspicion came from Drake Thomas of Anthropic's safety and alignment team: "If it raised our odds of surviving this by even 1%, I would burn my entire equity stake without hesitation, and I think a lot of colleagues across the industry would too. Genuinely — we're just scared. This isn't calculated marketing."

So the next chapter of this story isn't going to come from technology — it's going to come from process. If Congress answers the antitrust question in any form, the people who are each separately saying they're scared right now will finally have a path to sign the same document. If there's no answer, it will become clear whether OpenAI and Anthropic put out a joint proposal, or whether, as Schulman suggested, antitrust was just an excuse all along. METR's eight-week investigation will be the first test of how far an outside party can push back on a company's own incident report. METAL has covered the dispute over priority that broke out after OpenAI announced it had solved a famous open problem — a story from the same week that captures how the industry is simultaneously boasting about its capabilities and confessing to their dangers.

This isn't the first time someone has proposed hitting the brakes together. What's different now is that the people saying it are the ones holding the wheel. And the lawyer who jumped in between them still hasn't gotten an answer.