switch

SACRAMENTO — Gov. Gavin Newsom on Friday ordered the formation of a panel to come up with safety regulations for the state’s artificial intelligence companies, including the development of a possible “kill switch” for AI programs that go rogue.

Newsom issued an executive order to convene “a group of world-leading experts” to provide a “guide” for the state to bolster its laws around AI security, according to statement from the governor’s office.

The panel could put consider proposals for independent third parties to write safety plans or for companies to develop an emergency shutoff.

Newsom, who is considering a 2028 run for president, two years ago vetoed a state bill that would have mandated a kill switch.

Some Democratic and Republican leaders around the country are scrambling to come up with safety proposals after employees at AI companies recently warned that the technology could create catastrophe, including ending humanity.

A researcher for AI company Anthropic said he left the company over concerns that AI companies, including OpenAI, are “gambling with our lives” as they race ahead to improve AI that could surpass human intelligence.

The researcher, Jacob Coxon, said in a social media post: “People building AI earnestly believe that it could kill us all by the end of the decade.”

State Sen. Chris Cabaldon (D-West Sacramento), who chairs the state committee overseeing technology issues, told the Times he’ll hold hearings on artificial intelligence this fall. Those meetings were planned before the recent warnings about AI’s dangers, but will be informed by the events of the last few weeks, he said.

He said the state’s rules around AI don’t go far enough.

“We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity,” he said.

Any new laws could have a disproportionate effect on the global tech industry because many top AI companies are based in California.

A proposed law in 2024 by Sen. Scott Wiener (D-San Francisco) would have required tech firms to be able to turn off the AI models they directly control if things go awry.

Newsom, in his veto message at the time, cited concerns about stifling innovation. He also expressed concern that smaller AI models, which would have been exempt from the law, could also be dangerous.

An artificial intelligence “kill switch” which can be checked by a third-party may need to be mandatory for companies, a co-founder of one of the world’s largest AI firms has said.

Jack Clark, one of seven founders of Anthropic, said a way of shutting off AI software completely if it gets too dangerous was the kind of thing society “might want to eventually pass rules around”.

Clark said “most labs have different ways of being able to pull the plug”, including Anthropic, but said policymakers may need to enforce having one.

The rapid development of AI and fears over the risks that the technology poses to humanity have been thrust into the spotlight by a series of warnings from executives and staff at AI firms.

Some have said publicly there is a chance that the technology could, if unchecked, kill all humans.

Anthropic head Dario Amodei over the weekend called for the pace of AI development to slow and be more closely monitored, as the company has done before, though some have questioned the motivations behind this.

Amodei did not say how exactly how AI development could be slowed down and said any action to reign in AI development should be done “without sacrificing commercial advantage”.

Clark told the BBC that specifics around “kill switch” requirements and verification should be part of “the larger policy conversation” taking place around AI.

“Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?” he asked.

“I think that’s the kind of thing society is going to want to know and might want to eventually pass rules around.”

Anthropic, which was formed in 2021 by a group of former employees of its rival OpenAI, is currently at the centre of a debate around AI safety.

Last week, a post from an artificial intelligence researcher who quit Anthropic over concerns AI could wipe out humanity went viral.

In response, Anthropic scientist Evan Hubinger said he personally thought the possibility of human extinction from AI was “>10% within the next decade”.

Computer scientist and Nobel Prize winner Geoffrey Hinton, known as the “Godfather of AI”, told the BBC on Friday that a 10% chance of AI killing all humans was “not unreasonable”, external.

Asked what percentage he would put on all humans being killed by AI, Clark said: “I don’t think these statistics are that useful”, but added that allowing AI to continue as a “totally unregulated industry” was a bad idea.

“We are rolling dice with immense risks,” Clark said. “And the point is, we have to change the course of this industry.”