The MarginAnalysis

Can AI Be Stopped? What History Says It Takes

The people building AI have written down, with dates attached, when they would slow down or stop. Every one of those documents is still findable, and so is what happened next. Read them in order and then hold them against the four times in history a technology was actually halted.

Dark cover plate. A green Analysis chip, the numeral 3 set large in italic serif, and the line reading conditions stopped every technology that was stopped, AI has none. At right, a tally of four rows: Asilomar 3, Germline 2, CFCs 1, and a lit green row reading AI 0.

Not on its current terms. Every technology that was ever halted had at least one of three conditions: no revenue yet (Asilomar, 1975), a single government able to act alone (germline editing, 2019), or a substitute the incumbent could sell instead (CFCs, 1987). AI in 2026 has none of the three, and the people building it have said so in writing.

That last part is what makes this question unusual. It is the most heavily documented race in industrial history. The people running it have published, with dates, the exact conditions under which they would stop, and every one of those documents can still be read. What follows reads them in order, then sets them beside the four historical cases people reach for when they say a technology can be stopped.

The film reads each document on screen, in date order, over 43 minutes. It makes no claim about anybody's motives: no sentence in it begins with "because", since nobody can see inside anybody else's head. It shows what was committed to in writing and then what was done. This article follows the same rule and stands on its own.

Can AI development be stopped?

History says a technology stops only when at least one of three conditions holds: the money has not started flowing, one government can act alone, or the incumbent has a substitute it can profit from. Asilomar in 1975 had all three, germline editing had two, CFCs had one. AI in 2026 has none, which is why no written commitment has held.

Hold AI against each condition in turn. The money is flowing at a scale with no precedent: $852 billion for OpenAI in April 2026, $965 billion for Anthropic in May 2026, $250 billion for xAI when SpaceX bought it in February 2026, and that is three companies out of many. No single government can act alone, because the leading labs sit in at least two jurisdictions that are not coordinating. And there is no substitute. In every earlier case the incumbent had something else to sell, a different refrigerant or a containment cabinet. Here the product is the thing itself.

So understanding the risk was never the missing piece. The people named below understood it, in writing, with signatures. What was missing each time was one of the three structural conditions.

What did OpenAI's Charter promise?

OpenAI's Charter, published in April 2018, commits the organisation to stop competing with and start assisting any value-aligned, safety-conscious project that comes close to building AGI before it does. It names a typical trigger: a better-than-even chance of success in the next two years. The clause is still on OpenAI's site, and there is no public record of it ever firing.

OpenAI defines artificial general intelligence as highly autonomous systems that outperform humans at most economically valuable work. The Charter's long-term safety section names the fear directly: late-stage development becoming a competitive race without time for safety precautions. The same document describes the organisation's primary fiduciary duty as being to humanity.

What changed was the cost of keeping the promise. OpenAI was founded as a non-profit in December 2015. In 2019 it built a for-profit arm with investor returns capped at 100 times the money put in. Microsoft invested $1 billion that year, another $2 billion by 2022, and a further $10 billion announced in January 2023. By October 2025 the for-profit had become the main body, a public benefit corporation with the original non-profit holding 26%.

The valuations run: $157 billion in October 2024, $300 billion in April 2025, $730 billion in February 2026, $852 billion in April 2026. Handing your lead to a rival costs a charity its pride. It costs an $852 billion company almost everything.

What happened to the 2023 AI pause letter?

Nothing paused. The Future of Life Institute's open letter of 22 March 2023 asked every AI lab to immediately pause, for at least 6 months, the training of systems more powerful than GPT-4, and said governments should impose a moratorium if labs would not. It now carries 31,810 signatures. No lab paused for six months or six weeks.

The ask was narrower than it sounds. Training is the months-long, hundreds-of-millions-of-dollars process of building a model before anyone outside the company touches it. Nobody was being asked to switch off a product. They were being asked not to start the next one, eight days after GPT-4 had been released.

The signatories included Yoshua Bengio, Stuart Russell, Steve Wozniak, Yuval Noah Harari and Elon Musk. Six months from 22 March runs to about 22 September. On 12 July 2023, inside that window, the Financial Times and the Wall Street Journal both reported that Musk had launched xAI. That is 112 days after the letter. xAI shipped its first model that November, raised $6 billion at a $24 billion valuation in May 2024, reached $50 billion by December, and was valued at $250 billion when SpaceX bought it in February 2026.

The largest public attempt to slow the industry produced one more competitor, inside its own deadline.

Who signed the AI extinction risk statement?

The heads of all three leading labs signed it: Sam Altman of OpenAI, Demis Hassabis of Google DeepMind and Dario Amodei of Anthropic. The Center for AI Safety published the statement on 30 May 2023. It is one sentence of 22 words, with no demands, no deadline and no policy attached.

The sentence reads: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

Below the three chief executives sat the people who build the systems: Geoffrey Hinton, who had left Google that month to speak about the danger, Yoshua Bengio again, Ilya Sutskever, Mira Murati and John Schulman from OpenAI, Shane Legg and David Silver from Google DeepMind, Daniela Amodei from Anthropic, and Microsoft's chief technology officer Kevin Scott and chief scientific officer Eric Horvitz. Igor Babuschkin is listed as a co-founder of xAI, six weeks before the public was told xAI existed.

Then the money arrived. When Amodei signed, Anthropic had raised roughly $700 million in total. Amazon invested $1.25 billion that September and $2.75 billion the following March. Anthropic was valued at $61.5 billion in March 2025, $183 billion six months later, $380 billion in February 2026, and $965 billion when it raised $65 billion in May 2026. That is more than 1,300 times the money it had raised when its chief executive signed the sentence.

Do AI companies have a written rule for when to stop?

Yes, and it is real. OpenAI's Preparedness Framework, version 2, updated 15 April 2025 and 22 pages long, tracks three capabilities: biological and chemical weapons help, cyberattacks at scale, and AI self-improvement. For each, a model reaching the Critical threshold means halting further development until safeguards meet a Critical standard.

That is a specific written commitment to stop building, not just to delay a launch, made in advance by the company most often accused of having none.

The same document has a section 4.3, called Marginal risk. It says that if another developer releases a High or Critical capability system without comparable safeguards, and OpenAI can rigorously confirm it, OpenAI could adjust the level of safeguards it requires in that area. Three conditions are attached: the adjustment must not meaningfully increase overall risk, it must be publicly acknowledged, and OpenAI's safeguards must stay more protective than the rival's. The framework gives the reason for that last condition itself: avoiding a race to the bottom on safety.

That is not carelessness. It is what careful people write when being the only careful one is a losing position: a safety rulebook with a clause for relaxing the safety rules if a competitor relaxes theirs first.

Why did Google remove its AI weapons pledge?

Google's published reason was competition. On 7 June 2018 Sundar Pichai published Google's AI principles, including a list of applications Google would not pursue, among them weapons whose principal purpose is to injure people. The list was live on 2 February 2025. On 4 February 2025 Google announced updated principles, and by 1 March 2025 the list was gone.

The 2018 document came after a staff revolt over a Pentagon contract. Pichai called the principles concrete standards that would govern Google's research and products. The not-pursue list had four items: technologies likely to cause overall harm, weapons, surveillance violating internationally accepted norms, and technologies contravening international law and human rights. Directly beneath them sat one sentence saying the list might evolve as Google's experience deepened.

The 4 February 2025 post was written by James Manyika, a Google senior vice president, and Demis Hassabis. It explains the update by pointing to a global competition for AI leadership in a complex geopolitical landscape, and says democracies should lead AI development. The new page had three principles: bold innovation, responsible development and deployment, and collaborative progress. The announcement does not mention that the weapons line was removed.

This is not evidence that Google builds weapons, and no law was broken. Companies revise policies constantly. It is a written public promise not to do one specific thing, present one week and absent the next, replaced by a document whose stated reason is competition. Hassabis signed that post 20 months after signing the extinction statement.

Has any AI company ever held a model back?

One has, on the public record. Anthropic's "Core Views on AI Safety", published in March 2023, says it trained the first version of Claude in spring 2022 and chose to use it for safety research rather than release it, deploying only once the gap between Claude and the public state of the art had narrowed.

That is the only documented case of a lab voluntarily holding a finished model back, and the reason for releasing it is the important part. They shipped once shipping no longer moved the frontier. Stopping is survivable while you are behind. It costs you the company while you are ahead.

Other written commitments go further on paper. At the Seoul summit in 2024, twenty companies signed the Frontier AI Safety Commitments, including Amazon, Meta, Microsoft, Mistral, Samsung, NVIDIA, xAI and the Chinese lab Zhipu.ai. They commit not to develop or deploy a model at all if mitigations cannot keep its risks below their thresholds. That matters because the standard argument for not stopping is that China will not.

Compare the White House voluntary commitments of 21 July 2023, signed by seven companies: Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI. They cover security testing, sharing risk information, protecting model weights, watermarking and public reporting. None of the seven obliges anyone to pause or decline to release. Every line is about how to build, and none is about whether to build.

Anthropic's own Responsible Scaling Policy, first published in September 2023, has been revised nine times, reaching version 3.4 in July 2026. A policy that improves is a policy that changes, which is not damning by itself. It does mean the line moves.

Has any technology ever actually been stopped?

Yes, four times in the cases people usually cite, and each worked by a different mechanism. Recombinant DNA research paused itself for about 18 months in 1974 to 1976. Nuclear weapons were restrained by treaties 18 years after Hiroshima. Germline editing was stopped by a Chinese court in 2019. CFCs were phased out under the 1987 Montreal Protocol.

Asilomar. In July 1974 eleven of the field's most senior scientists, Paul Berg, David Baltimore, Herbert Boyer, Stanley Cohen and James Watson among them, called for a voluntary moratorium on the riskiest recombinant DNA experiments, in PNAS and in Science. In February 1975 about 140 people met at Asilomar in Pacific Grove, California, around 15% of them journalists. The moratorium held, work resumed under containment rules, and the National Institutes of Health made those rules formal in July 1976. There was no product and no revenue. The first commercial biotech company was founded two years after the letter.

Nuclear weapons. In June 1945 the Franck Report, from scientists at the University of Chicago, argued for a demonstration on a desert or barren island instead of use on a city, and predicted an arms race. On 17 July Leo Szilard's petition asking the President not to use the bomb without warning drew 70 signatures. Hiroshima followed on 6 August and Nagasaki on 9 August. What eventually worked came from governments: the Partial Test Ban Treaty in August 1963 and the Non-Proliferation Treaty in 1968, now with 191 states. The forecast during negotiations was 25 to 30 nuclear-armed states within 20 years. Today there are nine. The global stockpile peaked at about 70,300 warheads in 1986 and stands around 12,000, though the Federation of American Scientists says reductions have largely halted. Nuclear weapons never had a paying customer.

Germline editing. This is the case people cite most confidently, and it says close to the opposite of what they think. After He Jiankui announced gene-edited twins in November 2018, the field's own summit committee called his work irresponsible but explicitly did not call for a halt. It called for a rigorous, responsible pathway toward clinical trials. A March 2019 call in Nature for a global moratorium, signed by 18 scientists including Feng Zhang, Emmanuelle Charpentier and Paul Berg, was never adopted as binding. What stopped it was the Nanshan District People's Court in Shenzhen, which on 30 December 2019 sentenced He to three years in prison and a fine of 3 million yuan, about $434,000. Heritable editing was already illegal in more than 70 countries.

CFCs. In June 1974 Mario Molina and Sherwood Rowland showed in Nature that CFCs destroy stratospheric ozone. This was already a global industry. DuPont's chairman wrote to the US Senate as late as March 1988 that the evidence did not point to a need for dramatic reductions. In May 1985 Joe Farman, Brian Gardiner and Jon Shanklin published the Antarctic ozone hole. The Montreal Protocol followed in September 1987 and became the first universally ratified treaty in UN history, with 198 parties. The ozone layer is projected to recover to its 1980 state around 2040 across most of the world and by 2066 over Antarctica.

Why did the CFC ban work when industry was already dependent on it?

Because switching became more profitable than resisting. DuPont's patent on Freon expired in 1979, and by the mid-1980s DuPont had substitutes and the plants to make them, so a phase-out meant one product line replacing another. An EPA study in 1986 projecting 40 million extra US cancer cases made every year of delay a growing legal risk.

At that point the industry's most powerful opponent of regulation became its most useful advocate, and DuPont moved to support international controls faster than its European competitors. The argument did not change. The arithmetic did. Even so, it took 13 years from the 1974 paper to the 1987 treaty.

That makes CFCs the most important case for AI, because it is the only one stopped after the money was flowing. It shows dependence does not make stopping impossible. It also shows what it took: a substitute the incumbent could sell.

What would it take to stop AI?

One of two things that do not exist yet: a jurisdiction able to act alone, the role a Shenzhen court played for gene-edited children, or a substitute worth more to the labs than the thing itself, the role an expired patent played for DuPont. The first is a law nobody has written. The second is a business nobody has built.

Paul Berg, who organised Asilomar, wrote in 2004 that the Asilomar model would not work today, because such questions are now beset with economic self-interest and nearly irreconcilable ethical conflicts. The man behind the one clean example said it was a product of its conditions, and that the condition that had changed was money.

So the answer to why the race does not stop is not that nobody wanted it to. More than thirty thousand people said so. Three chief executives said so in 22 words. A charter said so with a trigger attached, and twenty companies signed a clause promising not to build at all past a threshold. The switch flips when stopping costs less than carrying on, and nobody in this race is near that point, because none of them has anything else to sell.

If you want the mechanism underneath the safety worry, the part where systems hit the score and miss the point, it is in reward hacking: why AI cheats the test to win. And if you want to know who, if anyone, is writing the rules instead, that is who regulates AI.