June 5 (UPI) — Artificial intelligence company Anthropic issued a warning about systems that can improve themselves and said that humans need a way to intervene when necessary.
AI systems will soon be able to better themselves — known as “full-recursive self-improvement” — and that has a lot of benefits, like for health care and science. But just like science fiction movies warn, it could cause serious risks to people, said Anthropic co-founder Jack Clark and leader of the Anthropic Institute Marina Favaro in a recent blog post.
“Full recursive self-improvement also might increase the risks of humans losing control over AI systems,” the blog said. “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.”
Clark called for the industry to give itself a “brake pedal” on CNN Thursday.
“When I look down at the car we’re driving, all I have is a gas pedal. I don’t have a brake pedal, and surely at some point in the future we might want that option,” he said. The inability to validate, verify and trust AI’s behavior is risky, he added.
Clark told CNN that countries have made similar changes in the past.
“We’ve done this before. In the height of the Cold War, under highly tense situations between rivalrous countries, they found ways to stabilize aspects of the nuclear arms race,” he said “All of this has been done before in other domains, and it may need to be something we do in the domain of AI.”
But critics say this talk of curbing AI is nothing new, even from Anthropic, which battled the Pentagon when it wanted full access to use its AI product.
In July 2025, Anthropic signed a $200 million contract with the U.S. Department of Defense. But CEO Dario Amodei said that Anthropic’s AI model Claude could not be used for mass surveillance in the United States or for autonomous weapons without human approval.
On Feb. 27, the Pentagon gave Anthropic a 5 p.m. deadline to comply with its demands that the government be able to use the service as it sees fit. Before the deadline, President Donald Trump announced that no government workers would be allowed to use Anthropic.
Then, Secretary of Defense Pete Hegseth labeled the company a supply-chain risk, which blocked it from any government contracts, but a judge struck it down in March.
“Anthropic might give the impression of being warm and fuzzy, but their definition of AI safety is narrow,” Steven Murdoch, a professor at University College London, told The Guardian. “Supporting U.S. authorities in the development of offensive capabilities has never been something they have spoken against.”
Murdoch said Anthropic’s blog left out evidence that AI is close to self-improvement.
“It is true that there’s some evidence that AI capabilities have increased and continue to increase with no limits becoming immediately clear,” he said. But, “I don’t think anything has fundamentally changed today that has caused Anthropic to publish this article.”
Murdoch pointed out that Athropic’s call for a pause on AI was similar to other proposals it has made in the past.
“It’s a reminder of what they are concerned about and have been concerned about for many years. I’m sure the attention is welcome, but again this isn’t a new thing,” Murdoch said. “Anthropic have been trying to get the attention of policymakers since they were founded.”