Anthropic urges AI pause, sees risk of losing control

Anthropic calls on major AI labs to consider a “coordinated and verifiable pause” in development, warning that rapid advances in technology could soon allow AI systems to develop themselves faster than society can manage the risks.

Claude’s creator said AI’s ability to complete tasks on its own is doubling approximately every four months and is trending toward “recursive self-improvement,” the point at which the technology can improve without human intervention.

“If systems are fully capable of creating their own successors, the ways we secure them, monitor them, and shape their behavior becomes much more important,” the startup said in a lengthy blog. to mail He added Thursday that a pause would allow society to “deal with the enormous consequences of this.”

“We’re not there yet, and iterative personal improvement isn’t inevitable. But it may come sooner than most organizations are prepared for,” Anthropic co-founder Jack Clark and Anthropic Institute leader Marina Favaro wrote in the post.

As technology becomes increasingly capable, fears have grown that advanced AI systems could slip out of human control and cause societal harm.

Anthropic’s own Mythos model sent shockwaves through industries including banking and software earlier this year with its ability to find vulnerabilities in existing code.

But regulation is moving slowly, especially in the United States, where many of the leading AI laboratories are located.

US President Donald Trump’s executive order earlier this week puts the burden on labs, asking them to voluntarily submit their most capable models for government cybersecurity testing before they are released to the public.

AI researchers have called for a pause before, but without much success.

Elon Musk, owner of AI lab xAI, was among the backers of a push by the nonprofit Future of Life Institute to pause AI development for six months in 2023 to give time for safety guardrails.

Anthropic has long positioned itself as a “security-focused AI laboratory.”

Earlier this year, its refusal to allow the US military to use its models for domestic surveillance and fully autonomous weapons sparked a backlash that led the government to put it on a national security blacklist that takes effect in late 2026.

Reuters reported on Friday that the dispute was showing signs of easing among some parts of the US government.

Still, Anthropic has continued to release increasingly powerful models, and in February backed away from a key security pledge, saying it would no longer hold back on potentially dangerous AI if it came close to matching its rivals’ capabilities.

It was recently valued at US$965 billion ($A1.4 trillion) in a major funding round and filed confidentially for an initial public offering in the US on Monday, putting it ahead of rival OpenAI both in valuation and in the race to secure key financing.

Anthropic’s post on Thursday warned that unilateral or poorly coordinated slowdowns could backfire and potentially reduce overall security if less cautious actors proceed.

A meaningful pause would require agreement among “numerous well-resourced laboratories” operating at technological frontiers, as well as rules on what conditions would trigger or lift such a pause and who would oversee it, it said.

“In contrast, a unilateral pause by a laboratory could be achieved immediately but would yield much less results: it would change who the frontrunner is, but would not create the broader negotiation process that is currently missing,” the initiative said.

Its research arm, the Anthropic Institute, plans to study the systems needed to support the slowdown and will bring together policymakers, researchers, civil society groups and rival AI firms in the coming months to discuss managing risks such as duplicative self-improvement.

OpenAI, xAI, ​Alphabet, Meta Platforms and French firm Mistral did not immediately respond to requests for comment on whether they would join the call.

Latest stories from our writers