Anthropic urges AI pause, sees risk of losing control

Anthropic urges major AI labs to consider a coordinated and verifiable pause in development, warning that rapid advances in technology may soon allow AI systems to develop themselves faster than society can manage the risks.

Claude’s creator said AI’s ability to complete tasks on its own is doubling about every four months and is heading toward “recursive self-improvement,” the point at which the technology can improve without human intervention.

“If systems are fully capable of generating their own successors, the ways we secure them, monitor them, and shape their behavior become much more important,” the initiative said in a lengthy blog post on Thursday, adding that a pause would allow society to “cope with their enormous impact.”

“We’re not there yet, and iterative personal improvement isn’t inevitable. But it may come sooner than most organizations are prepared for,” Anthropic co-founder Jack Clark and Anthropic Institute leader Marina Favaro wrote in the post.

None of this guarantees that recursive personal growth is on the horizon. It remains to be seen whether Claude has the research talent to choose the right problems to work on. But if these trends continue, it is plausible that AI systems will design and build their own successors. This…— Antropik (@AntropikAI) June 4, 2026

As technology becomes increasingly capable, fears have grown that advanced AI systems could slip out of human control and cause societal harm.

Anthropic’s own Mythos model sent shockwaves through industries including banking and software earlier this year with its ability to find vulnerabilities in existing code.

But regulation is moving slowly, especially in the United States, where many of the leading AI laboratories are located.

US President Donald Trump’s executive order earlier this week puts the burden on labs, asking them to voluntarily submit their most capable models for government cybersecurity testing before they are released to the public.

AI researchers have called for a pause before, but without much success.

Elon Musk, owner of artificial intelligence lab xAI, was among supporters of a push by the nonprofit Future of Life Institute to pause AI development for six months in 2023 to give time for safety guardrails.

Anthropic has long positioned itself as a security-focused AI laboratory.

Earlier this year, the US military refused to allow it to use its models for domestic surveillance and fully autonomous weapons, prompting a government backlash that put it on a national security blacklist that takes effect in late 2026.

Reuters reported on Friday that the dispute was showing signs of easing among some parts of the US government.

Still, Anthropic has continued to release increasingly powerful models, and in February backed away from a key security pledge, saying it would no longer hold back on potentially dangerous AI if it came close to matching its rivals’ capabilities.

It recently valued US$965 billion ($A1.4 trillion) in a major funding round and filed confidentially for an initial public offering in the US on Monday, putting it ahead of rival OpenAI both in valuation and in the race to secure key funding.

Anthropic’s post on Thursday warned that unilateral or poorly coordinated slowdowns could backfire and potentially reduce overall security if less cautious actors proceed.

A meaningful pause would need to be agreed upon among “a large number of well-resourced laboratories” operating at technological frontiers, it said, as well as rules for what conditions would trigger or lift such a pause and who would oversee it.

“In contrast, a unilateral pause by a laboratory could be achieved immediately but would yield much less results: it would change who the frontrunner is, but would not create the broader negotiation process that is currently missing,” the initiative said.

Its research arm, the Anthropic Institute, plans to study the systems needed to support the slowdown and will bring together policymakers, researchers, civil society groups and rival AI firms in the coming months to discuss managing risks such as duplicative self-improvement.

OpenAI, xAI, Alphabet, Meta Platforms and French firm Mistral did not immediately respond to requests for comment on whether they would join the call.