A controversial bill that would require developers of advanced AI models to adopt safety measures is one step closer to becoming law.

The bill, SB 1047, would require developers of future advanced AI models to create guardrails to prevent the technology from being misused to conduct cyberattacks on critical infrastructure such as power plants.

Developers would need to submit their safety plans to the attorney general, who could hold them liable if AI models they directly control were to cause harm or imminent threat to public safety.

The bill, introduced by Sen. Scott Wiener (D-San Francisco), passed the state Assembly on Wednesday, with 41 votes in favor and nine opposed. On Thursday, the measure was approved by the state Senate in a concurrence vote. It now heads to Gov. Gavin Newsom’s office, though it’s unclear whether Newsom will sign or veto the bill.

“Innovation and safety can go hand in hand — and California is leading the way,” Wiener said in a statement.

A spokesperson for Newsom said the governor will evaluate the bill when it reaches his desk.

Wiener’s bill was fiercely debated in the Bay Area’s tech community. It received support from the Center for AI Safety, Tesla CEO Elon Musk, the L.A. Times editorial board and San Francisco-based AI startup Anthropic.

But it was opposed by Democratic congressional leaders as well as prominent AI players including Meta and OpenAI, who raised concerns about whether the legislation would stifle innovation in California.

Democratic congressional leaders including former House Speaker Nancy Pelosi, Rep. Ro Khanna (D-Fremont) and Zoe Lofgren (D-San Jose), have also opposed the bill and urged Newsom to veto it. They argue the legislation could hurt California’s growing AI industry, home to ChatGPT maker OpenAI, and cite efforts Congress is making related to AI.

“There is a real risk that companies will decide to incorporate in other jurisdictions or simply not release models in California,” Khanna, Lofgren and six other Democratic congressional representatives wrote in a letter to Newsom.

“While we want California to lead in AI in a way that protects consumers, data, intellectual property and more, SB 1047 is more harmful than helpful in that pursuit,” Pelosi said in a statement.

Wiener and other legislators supporting the bill disagree, contending it would foster innovation while also protecting the public.

“You have to put guardrails,” said Assemblymember Devon Mathis, (R-Visalia), before the Assembly’s vote on Wednesday afternoon. “We have to make sure they are going to be responsible players.”

Proponents of SB 1047 say it requires developers to be responsible for the safety of advanced AI models in their control, which could help prevent catastrophic AI events in the future.

“I worry that technology companies will not solve the significant risks associated with AI on their own because they’re locked in their race for market share and profit maximization,” said Yoshua Bengio, a professor at Université de Montréal, and the founder and scientific director of Mila — Quebec Artificial Intelligence Institute at a press briefing this week. “We simply can’t let them grade their own homework and hope for the best.”

Backers also say AI should be regulated similar to other industries that pose potential safety risks.

“This is a tough call and will make some people upset, but, all things considered, I think California should probably pass the SB 1047 AI safety bill,” Musk wrote on X on Monday. “For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public.”

Earlier this month, the bill passed a key state Senate committee after Wiener made significant changes, including removing a perjury penalty and changing the legal standard for developers regarding the safety of their advanced AI models.

San Francisco-based AI startup Anthropic’s CEO Dario Amodei said he believed the bill’s “benefits likely outweigh its costs” in an Aug. 21 letter to Newsom. The letter did not endorse the bill, but shared the company’s viewpoint on the pros and cons.

“We want to be clear…that SB 1047 addresses real and serious concerns with catastrophic risk in AI systems,” Amodei wrote. “AI systems are advancing in capabilities extremely quickly, which offers both great promise for California’s economy and substantial risk.”

But some tech companies like OpenAI said they opposed the bill even after the changes.

“The broad and significant implications of AI for U.S. competitiveness and national security require that regulation of frontier models be shaped and implemented at the federal level,” OpenAI’s Chief Strategy Officer Jason Kwon wrote in an Aug. 21 letter to Wiener. “A federally-driven set of AI policies, rather than a patchwork of state laws, will foster innovation and position the U.S. to lead the development of global standards.”

Wiener said he would welcome a strong federal AI safety law that preempts his bill.

“If past experience is any indication, enacting such a (federal) law will be an uphill fight,” Wiener said in a statement. “In the meantime, California should continue to lead on policies like SB 1047 that foster innovation while also protecting the public.”

SB 1047 is among roughly 50 AI-related bills that addressed various aspects of the technology’s impact on the public, including jobs, deepfakes and safety in the state Legislature.