Whose Breathing Room?
The headlines took one line from Dario Amodei's essay on Saturday: the AI labs must slow down. The passage I keep going back to sits further in. Part of slowing down safely, Amodei writes, is keeping the democratic world's lead over autocracies as wide as possible, so the labs have the "breathing room" they need. The ways to protect that lead: no advanced chips or chipmaking equipment for China, a crackdown on chip smuggling and on remote access to data centres outside China, and a crackdown on unauthorised distillation by companies in authoritarian countries. Done well, he estimates, this widens America's lead significantly over the next three to five years.
Keep that paragraph in mind, because the week that produced the essay was about something else entirely, at least on the surface: the end of the world.
Two days before the essay, Elon Musk looked at the story and called it a setup. On the All-In podcast, David Sacks went further and described a coordinated operation by the safety movement whose real target is open source. His evidence was speed. Within about fifteen minutes, Encode AI, the AI Policy Network and the AI Futures Project had all amplified the same post, and Sacks says all three are funded by Jaan Tallinn, who co-led Anthropic's Series A. On Saturday, Musk posted three words agreeing with Amodei.
The post itself went up on Tuesday. Jacob Coxon had spent three years on pretraining at OpenAI and Anthropic. He left Anthropic four months after joining, two months before his equity would have vested, and wrote that neither company was acting responsibly and that the people building AI earnestly believe it could kill everyone before the decade is out. It passed 160 million views. Evan Hubinger, who leads alignment science at Anthropic, backed him in public and put his own estimate above ten percent within a decade. Others followed with their numbers. Geoffrey Irving, former chief scientist of the UK's AI Security Institute, says around fifty percent. Marcus Williams at OpenAI says seventy percent within three years unless there is regulation or a slowdown. His colleague Ted Sanders puts the chance for the next decade at essentially zero. Jan Leike counts 1,386 people at frontier AI companies who have signed a statement asking for the option to pace development.
I think the fear is real, and I don't think it's a close call. Coxon walked away from money. Hubinger said what he said with an IPO coming. On the same podcast where Sacks called it an operation, Chamath Palihapitiya pointed out that Anthropic will now have to address these beliefs in its filing with the SEC, and nobody writes the possible extinction of humanity into their risk factors as a sales tactic. Zvi Mowshowitz, who has followed these researchers for years, makes the plain point that the commercial incentive is to talk the risk down. For the setup theory to hold, hundreds of people would have to fake a belief that costs them.
And still, go back to the paragraph at the top.
The one thing Anthropic commits to on its own is letting outside evaluators such as METR work inside the company, with desks, badges, employee-level access and the right to publish what they find. That's a real concession and I don't want to wave it away. The slowdown itself, though, comes with a condition. Slow down by more than your lead, Amodei writes, and projects tied to the Chinese Communist Party pull ahead. Safety gets measured in units of lead. The instruments for protecting the lead are chip controls and a distillation crackdown, and by his own account those are the two things that let Chinese labs keep up. Agreements with Beijing come last, starting with narrow red lines such as using AI to build biological weapons.
None of this is new for him. In January 2025, days after DeepSeek's R1 knocked hundreds of billions off Nvidia's market value, he argued that export controls decide whether we end up with one AI superpower or two, and that a temporary lead can be turned into a lasting one because AI helps build better AI.
What's missing from Saturday's essay is the phrase open weights. After this summer, that silence says a lot. On July 24, 77 companies, Nvidia, Meta, Google, Microsoft, OpenAI, Hugging Face and Mistral among them, signed a letter defending open-weight models and warning against bans on distillation. Jensen Huang pushed it hard on X. Reporting at the time said some officials believed Anthropic was lobbying against Chinese open models to get rid of competition. Three days later Amodei replied that Anthropic has never advocated a ban, that open models without dangerous capabilities are a public good, and that yes, a ban would shield American AI companies from competition, which he says was never his goal. What he wants instead is the same set as in the essay: chip controls, action against industrial-scale distillation, and mandatory safety testing for every sufficiently capable model, open or closed. Mike Masnick at Techdirt answered that the distillation part adds nothing to safety and mostly hobbles cheaper competitors.
OpenAI signed that letter, by the way. On Saturday Sam Altman also said OpenAI would match Anthropic's evaluator commitment. I haven't decided whether that's a contradiction or just a measure of how little either document asks of anyone.
So, fear of the end of the world, or fear of China? This is where I get stuck.
The extinction worry and the commercial worry recommend the same things. If you believe a misaligned system could end civilisation, you have honest reasons to dislike open weights: once weights are out on the internet, nobody can patch them or pull them back. You want a few labs, closely watched, well ahead of any actor you can't negotiate with. If your worry is a Chinese model at a fraction of your price, you want the identical list. Vercel's figures from June show open-weight models carrying about 29 percent of the tokens through its gateway and under 4 percent of the spend. Both motives write the same policy, so the policy can't tell you which one held the pen.
They only come apart at the edges, and I can see three.
The first is whether the labs would slow down if China never joins. Amodei's answer is no, not beyond the size of the lead, and at least he says so openly.
The second is distillation. Distilled models trail the frontier almost by definition, so it's hard to see who is safer once they're stopped. Amodei would answer that they narrow the gap, and a narrower gap means less breathing room. That exchange shows what bothers me most about his framework: the lead argument can turn almost any measure against competitors into a measure for safety.
The third is who gets to coordinate. The essay asks Washington for a narrow antitrust waiver so the frontier labs can discuss pacing among themselves. Market leaders agreeing among themselves with the state's permission is exactly what competition law was invented to prevent. It can be a safety mechanism and a moat at the same time.
None of that makes Coxon a fraud or Amodei a cynic. I suspect most of them believe every word. Sincere people can arrive at a policy that protects their business without ever choosing to.
The other side isn't disinterested either. Nvidia, whose CEO promoted the July letter, loses revenue every time chip controls tighten, and earlier this month Huang declared the race to AGI over, a claim Gary Marcus took apart because the term was never even defined.
Europe barely exists in any of these documents. In the essay the world has democracies, which in practice means America, and autocracies, which means China. Mistral appears this summer as one signature on a letter. Chinese state-affiliated media, for their part, have made it a condition of the September talks that both governments first agree on what AI safety even means.
Our own rulebook complicates Masnick's argument, though. Under the EU AI Act, models with systemic risk already have to be evaluated, and publishing the weights doesn't exempt them; the Commission's enforcement powers started last month. Mistral keeps releasing open models anyway. So mandatory testing for open models is not automatically a ban in disguise. What would be new in the American version is everything packed around the testing.
I'm not a neutral reader of any of this. Trusq runs on Mistral, and for customers whose data can't go to an outside AI provider, we're designing around an open-weight Mistral model that we operate ourselves. Without open weights, that option doesn't exist. Nothing on the table today would touch a model that small. What would reach us is slower: a world where the models anyone can run are held a comfortable distance behind the ones only a handful of companies can sell.
Musk needed two days to get from calling this a setup to agreeing with Dario. I've had the same two days, and I can't tell you which of his posts was right.
#AI #OpenSource #AISafety