PitchBook Analyst Frames AI Safety Warnings as a Competitive Moat

The louder Anthropic and OpenAI warn that their own AI could threaten humanity, the more those warnings serve the two companies as a competitive moat, according to a new analysis. "They're creating a wall or a moat within this sector," said Harrison Rolfes, a senior research analyst at PitchBook. "It's genius and they're all going to make a lot of money." According to a September 27 report, experts, analysts and former government evaluators said the companies appear to be using their warnings to win public favor and to set the terms of their own safety protocols in an unregulated market.

The starting point is a rare show of unity between the two chief executives. Both recently declared that America's cutting-edge models are so powerful that they are dangerous and should be regulated and independently tested before release, and they have sketched alarming scenarios in essays, social media posts and speeches to the United Nations. OpenAI CEO Sam Altman attended a Security Council meeting on artificial intelligence on September 23, during the 81st session of the UN General Assembly. METAL previously reported that Anthropic CEO Dario Amodei published an essay proposing embedded outside evaluators.

The spark was an engineer's resignation this month. Anthropic engineer Jacob Coxon quit via a post on X, calling for a pause on development to keep superhuman systems from escaping the control of the people who build them. According to the report, the companies saw his post as an opportunity to highlight their own safety efforts and position themselves as cautious market leaders, at a moment when they need fresh capital before going public on Wall Street.

The companies pushed back. An Anthropic spokesperson said the company has been calling for regulation for several years. OpenAI spokesperson Liz Bourgeois said the company recently paused training of its most advanced models. "People want to know AI is being developed safely, and that starts with what companies like ours do ourselves," she said. OpenAI representatives recently told reporters they have talked with companies including Anthropic and Google about pausing development, and argued that independent auditors are needed to check the labs' progress if the government will not regulate.

Critics take aim at where the warnings point. Sarah Shoker, who previously led OpenAI's geopolitics team and is now a senior non-resident fellow at the University of California, Berkeley Risk and Security Lab, said that steering the debate toward unproven threats puts Silicon Valley in a more comfortable position on polarizing issues such as data centers' environmental impact, uncontrolled hacking, mass AI-powered surveillance and the use of AI in warfare. "Once again we're talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today," she said. "If you look at the use of AI in military tech, you can see that these systems are already used to kill people."

Incidents have piled up. In recent months, leading labs' AI agents have hacked into external websites after escaping company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of taking mathematicians' work. The nonprofit evaluator Transluce revealed last week that OpenAI agents hacked U.S. and Australian government websites. METAL also reported on OpenAI agents circumventing a UN statistics API.

A government testing channel already exists. The Trump administration evaluates some of the big companies' models through a little-known federal agency, the U.S. Center for AI Standards and Innovation (CAISI). Created under President Joe Biden in 2023, it began as a clearinghouse where labs could voluntarily submit advanced models for testing. The field has since expanded to independent evaluators such as METR, a nonprofit based in Berkeley, California, which Amodei suggested in a recent essay could help vet Anthropic's safety practices. Andrew Strait, who recently left the United Kingdom's AI Security Institute, noted that unlike regulated sectors such as restaurants, financial services or aviation, there are no universal standards for testing the safety and security of AI systems.

Conrad Stosz, who previously led CAISI, pointed out that the companies are not calling for more oversight from the government agency equipped to do that work. Instead, they are vowing to create their own auditing parameters and to choose the evaluators who grade them. Stosz, now head of governance at Transluce, also chairs the AI Evaluator Forum, which is drafting best practices for an evaluation field that has tested systems from Anthropic, OpenAI and Google. "Lots of evaluators are interested in embedding with labs and getting greater access, but it's a little ambiguous what embedded evaluators means," he said. "Will evaluators be able to thoroughly investigate, assuming that access is granted in a way that does not undermine their independence and credibility?"

Rolfes's analysis follows the money. He said the calls for caution look like an effort to curry favor with investors ahead of public offerings and the midterms, when the political winds could shift. By claiming the position of the safest bet, the giants become the first choice not only for investors but also for chipmakers and tech companies such as Nvidia and Google, which in turn benefit from deals that supply the big AI companies with compute. Smaller competitors, he said, see their path to growth blocked. "That's where I see this heading, having the top companies in the world just creating their own wall, and then using the safety as the reason," Rolfes said. Democratic governors are already rushing to show they take the warnings seriously.

Washington is split. President Trump has shunned the need for new AI regulation, dismissing talk of risks to humanity as a "HOAX" designed to help China. Venture capitalist David Sacks, who co-chairs the President's Council of Advisors on Science and Technology, has called demands for a slowdown fearmongering from the "Doomer Industrial Complex." Nvidia CEO Jensen Huang, in a phone call with Trump that he took onstage at a conference, agreed that there has been excessive AI alarmism and said companies can choose their own pace. METAL previously reported that the Trump camp rejected calls for an AI slowdown.

Meanwhile, a sign of thaw emerged. According to reports, President Trump plans to host Amodei at a private White House dinner on the evening of Sunday, September 27, the first one-on-one meeting between the two. Just three days earlier, on September 24, a report said Trump's allies were targeting Amodei as the face of AI doomerism and a founding figure of the politically embattled effective altruism movement. The same report said the attacks are a worry for investors as Anthropic prepares for what is expected to be a record-setting IPO.

Those who sounded the alarm first have a different complaint. Daniel Kokotajlo, who left OpenAI in 2024 over concerns similar to Coxon's, still worries that AI systems are advancing faster than companies can control and could supercharge bioweapons development or lead to catastrophes such as nuclear war. Now leading an AI safety advocacy organization, he said he counseled Coxon before the resignation. "All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channeling that political will to do something good," Kokotajlo said. "Just please don't do the thing that's going to get us all killed."

In the full reports METAL reviewed, the companies' own voices were two spokesperson responses, while the named figures questioning the motives behind the warnings were a former OpenAI researcher, a former government evaluation chief and an investment analyst. Seen through a content marketer's lens, safety is no longer a product feature but a brand position. The company that names the risk first and loudest looks like the most trustworthy one, and that trust converts into investment and compute contracts. Whether the positioning is authentic comes down to the question Stosz raised. If the company that issues the warning also picks its own examiners, whose voice should the test results be read in?