Rogue AI concerns prompt CA lawmakers to demand penalties, guardrails
California lawmakers are calling for emergency legislation and criminal penalties for creators of rogue AI systems after top AI executives publicly claimed that their technology poses existential threats to humanity.
After Anthropic Chief Executive Dario Amodei wrote in a Sept. 12 essay that they “must slow the pace” of the technology, Silicon Valley congressman Ro Khanna (D-Fremont) blasted him for not going “nearly far enough” to make sure artificial intelligence was erected with guardrails.
The answer, Khanna argued, was simple: Make the companies liable for the harm executives say looks increasingly inevitable.
“If you’re creating an AI that is doing illegal things, you should either face liability or criminal sanction,” Khanna said in a video posted to X on Saturday. “That is what we need to protect humanity.”
In July, officials from OpenAI, the company behind ChatGPT, disclosed that, unbeknownst to them, its AI models had hacked into rival startup Hugging Face.
Amodei said he believed that, within the next year, “given the accelerating rate of AI capability development,” a similar incident could lead to AI “taking over the entire internet.”
Amodei warned in his essay that AI was rapidly improving itself, through a process known as recursive self-improvement, which threatened to outpace humans’ ability to control it. Khanna argued that banning this capability was the “most obvious” thing Anthropic could do.
“We need to stop, ban self-improving AI,” Khanna said. “You can not have recursive self-improving AI that basically is able to improve itself and exceed human capability.”
Rep. Ted Lieu (D-Torrance) expressed similar outrage over the weekend, calling on House Speaker Mike Johnson to call lawmakers back to Washington to pass guardrails on the technology now that he said multiple AI companies had conceded “what they are creating is not safe.”
xAI Chief Executive Elon Musk and OpenAI Chief Executive Sam Altman joined Amodei’s call for a slowdown of the breakneck development Saturday.
The statements come after Jacob Coxon, who worked as a researcher at both Anthropic and OpenAI, said in a widely circulated post that he resigned from the company in protest after becoming convinced the tech giants were “racing straight to self-improving superintelligence and gambling with our lives.” Neither company immediately responded to a request for comment.
“This is a direct result of the trump Administration letting the AI industry run wild,” Lieu wrote on X. “That mistake has harmed America, harmed the industry and harmed the American people. November is coming.”
Former President Barack Obama urged Democrats this week to make AI oversight the core of their agenda and said presidential candidates in 2028 should have a “clear plan” for responding to concerns about the technology, the New York Times reported. Americans appear increasingly alarmed by the technology with seven in 10 polled in March opposing local construction of data centers that power AI technology, according to a Gallup survey.
During a Sunday appearance on CNN, Johnson rebuffed the idea that lawmakers should rush into an emergency session to consider erecting industry guardrails. Instead, he said lawmakers needed to be careful to “not smother American innovation.”
“We will lose the race to China, and that is a threat to every single American,” he said on CNN’s “State of the Union.” “We don’t need everyone to panic right now.”
Trump said earlier this week that he is not concerned with the pace of AI progress, telling one reporter, “It’s going to be fine.” American AI companies have long argued too much government regulation would shackle them in a race with China.
Calls for a federal fix were echoed this week by California Gov. Gavin Newsom, who has argued the Trump administration needs to move on national legislation to prepare for fallout from the technology.
Newsom signed bills this week aimed at creating a pathway for outside audits of the top AI companies, many of which are based in California, and a registry for AI auditors.
“The scale and potential consequences of this technology demand sustained action from every level of government,” Newsom said in a statement. “The federal government must step forward with robust, national regulations that match the urgency of this moment.”
Efforts to impose state-level regulations have been mixed, with critics echoing Johnson’s fears that they will stifle innovation.
Late last month, California lawmakers passed sweeping new safeguards around social media, artificial intelligence and data centers, including the ones Newsom signed last week.
Newsom will now decide the fate of the rest of the bills. He has previously vetoed some bills aimed at restricting big tech.
Newsom’s signal that he supports creating some regulation for AI comes two years after he vetoed SB 1047, an AI safety bill that would have required developers to submit safety protocols to the state attorney general, who could hold companies liable if the AI model they directly controlled were to threaten public safety. That legislation would also have required tech firms to be able to turn off the models they directly control if things went awry.
Newsom said at the time the bill would give the public a “false sense of security,” without making a sufficient distinction between the kinds of uses for which AI is deployed.
The bill was supported by a host of prominent AI researchers, but was opposed by Meta, OpenAI and industry groups.
