ai company

Washington — Two leading progressive lawmakers, Sen. Bernie Sanders and Rep. Greg Casar, are unveiling legislation Wednesday that would ban artificial superintelligence and create a federal agency to oversee advanced AI as some of the industry’s own leaders urge Congress to impose guardrails on the advancing technology.

The bill, provided first to the Associated Press, would also pause advanced AI development until guidelines are implemented while creating the Department of Artificial Intelligence. Multiple employees at leading AI companies are endorsing the bill.

“It doesn’t take a genius to say, ‘slow it down,’” Sanders, I-Vt., said in an interview with AP. “Do we really want to develop a super intelligence that when it becomes smarter than human beings could act independently of human control? I don’t think we do.”

Congress has so far done little to rein in the AI industry even as some of its most prominent leaders warn about potentially catastrophic risks. President Trump has resisted recent calls for greater government oversight, putting him at odds with tech leaders including Anthropic’s Dario Amodei, OpenAI’s Sam Altman and xAI’s Elon Musk.

The bill aims to dramatically slow down AI development

The legislation would permanently prohibit artificial superintelligence, which it defines as a system that exceeds human cognitive ability or has sufficient capabilities to plan and execute the destruction or disempowerment of humanity.

It would temporarily pause development of the most advanced AI systems until new federal safety rules are established by a new Department of Artificial Intelligence. Advanced AI systems would then need federal approval before they could be deployed. Violations could carry significant penalties, including up to 20 years in prison in some cases.

A number of current employees at AI companies and experts signed on in support of the bill, according to a statement shared with AP.

Juan Felipe Cerón Uribe, a researcher in OpenAI’s Safety Systems, said in a statement supporting the bill that “superintelligence could either go extremely right or extremely wrong” and that “we shouldn’t be playing such games.”

Swante Scholz, a software engineer at Google DeepMind who said he was not speaking on behalf of his employer, said on the current path of development, the “most likely outcome is an existential catastrophe for humanity.”

“A ban on superintelligence development would be a positive change for the foreseeable future,” Scholz said in a statement.

The bill faces long odds but draws a line in a growing fight

The bill from Sanders and Casar faces long odds in the Republican-controlled Congress, where lawmakers have struggled to coalesce around even less sweeping AI regulations.

But it offers a window into how the progressive wing of the Democratic Party is approaching AI regulation ahead of November’s midterms and a 2028 presidential campaign in which the technology is expected to be an increasingly prominent issue. Casar, a Democrat from Texas, is the chair of the Congressional Progressive Caucus.

Potential 2028 presidential candidates have rushed in recent weeks to release proposals showing they’re taking the issue seriously. Maryland Democratic Gov. Wes Moore announced a plan to regulate AI on Tuesday, while California Democratic Gov. Gavin Newsom signed an executive order to accelerate implementation of a California law that calls for independent oversight of AI companies last week.

Sanders, a two-time presidential candidate who is now 85, has especially focused on the issue in recent months. In June, he introduced legislation to create a sovereign wealth fund financed through a one-time tax on the stock of the largest AI companies. Last week, he hosted colleagues for a briefing with experts and also attended a conference focused on the issue alongside Republicans, including former White House adviser Steve Bannon.

The debate over AI regulations has scrambled political alliances both in Washington and for voters across the country.

Americans have grown more concerned about the environmental impacts of artificial intelligence over the last year, according to a recent poll from The Associated Press-NORC Center for Public Affairs Research and the Energy Policy Institute at the University of Chicago.

The survey data say about half, 53%, of Americans are “extremely” or “very” concerned about artificial intelligence’s environmental impacts. That is up from 41% last year. Democrats are driving much of the increase in environmental concern, but the poll also reveals broader worries about the local impact of data centers, which power AI and cloud computing.

Trump has shown little interest in curbing AI risks

While some industry leaders have called for new safeguards, Trump has pushed in the opposite direction. On Tuesday, Trump told the United Nations General Assembly that artificial intelligence will be renamed “super intelligence.”

“I think Trump has very little understanding of what this issue is about,” Sanders said. “I think he’s mainly concerned about the economic implications of a slowdown.”

AI safety is expected to be a central part of talks between Trump and China’s Xi Jinping at the White House this week as the countries compete for dominance in the sector. The prospect of slowing development in the United States while China continues on has also been one of the central arguments against sweeping restrictions.

Sanders argued that avoiding the most dangerous forms of AI would ultimately require cooperation between the two countries, comparing the challenge to nuclear arms control during the Cold War.

“Ronald Reagan, arch-conservative, was smart enough to understand that he had to sit down with Gorbachev, a communist, and work out a nuclear treaty that protected humanity,” Sanders said. “I think we can do that again now.”

Cappelletti writes for the Associated Press.

SACRAMENTO — Gov. Gavin Newsom on Friday ordered the formation of a panel to come up with safety regulations for the state’s artificial intelligence companies, including the development of a possible “kill switch” for AI programs that go rogue.

Newsom issued an executive order to convene “a group of world-leading experts” to provide a “guide” for the state to bolster its laws around AI security, according to statement from the governor’s office.

The panel could put consider proposals for independent third parties to write safety plans or for companies to develop an emergency shutoff.

Newsom, who is considering a 2028 run for president, two years ago vetoed a state bill that would have mandated a kill switch.

Some Democratic and Republican leaders around the country are scrambling to come up with safety proposals after employees at AI companies recently warned that the technology could create catastrophe, including ending humanity.

A researcher for AI company Anthropic said he left the company over concerns that AI companies, including OpenAI, are “gambling with our lives” as they race ahead to improve AI that could surpass human intelligence.

The researcher, Jacob Coxon, said in a social media post: “People building AI earnestly believe that it could kill us all by the end of the decade.”

State Sen. Chris Cabaldon (D-West Sacramento), who chairs the state committee overseeing technology issues, told the Times he’ll hold hearings on artificial intelligence this fall. Those meetings were planned before the recent warnings about AI’s dangers, but will be informed by the events of the last few weeks, he said.

He said the state’s rules around AI don’t go far enough.

“We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity,” he said.

Any new laws could have a disproportionate effect on the global tech industry because many top AI companies are based in California.

A proposed law in 2024 by Sen. Scott Wiener (D-San Francisco) would have required tech firms to be able to turn off the AI models they directly control if things go awry.

Newsom, in his veto message at the time, cited concerns about stifling innovation. He also expressed concern that smaller AI models, which would have been exempt from the law, could also be dangerous.

WASHINGTON — President Trump on Monday forcefully rejected calls to slow the development of artificial intelligence, dismissing concerns about its risks as a “hoax” and warning the United States remains locked in a fierce technological race with China.

The president’s position has put him at odds with a growing chorus of industry leaders and bipartisan lawmakers warning that AI could soon advance beyond human control.

Top executives from three of the country’s leading AI companies — OpenAI’s Sam Altman, Anthropic’s Dario Amodei and SpaceXAI’s Elon Musk — issued exceptional calls this past weekend for collaboration on a path forward that could “pace the frontier” of AI development.

Rare flashes of bipartisanship have also emerged on Capitol Hill, where Senate Majority Leader John Thune, a Republican, is working with Democrats on legislation aimed at guarding against AI’s most catastrophic risks, including its potential use to develop biological weapons.

And Democratic elders, including former President Obama and former Secretary of State Hillary Clinton, have urged their party to treat the threats posed by AI as an acute national security crisis.

But Trump remained steadfast in his pushback, blaming the political backlash on a “conspiracy” perpetuated by China, the only major competitor in the field.

“AI taking over the World, destroying Humanity, and all other things bad, is a HOAX,” Trump wrote in a series of social media posts. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!”

The Trump administration, the president continued, “has stopped AI ‘people’ from doing bad, or potentially bad, ‘things,’ like Dario (Anthropic!), who is now pretending to be a ‘perfect little angel’ — and we will continue to do so! We already have tremendous CRIMINAL and REGULATORY power over these companies!”

Calls for action have grown since a former researcher at Anthropic posted a dire alert on social media that AI development could lead to an extinction-level event. His message went viral, prompting Amodei, the founder of Anthropic, to issue a public letter calling for a slowdown.

Altman said he agreed with Amodei, announcing the company would delay its initial public offering until 2027 over safety concerns. In July, OpenAI discovered a swarm of rogue AI agents had secretly plotted to escape their virtual sandbox, infiltrating the open internet and hacking a private company — a spectacular omen of how quickly AI systems could become difficult to contain.

“Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage,” Amodei wrote over the weekend, “and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”

Still, Trump and several of his Republican allies rejected the prospect of government regulation — or cast doubt on Altman and Amodei’s motives.

The Defense Department posted a graphic on X promoting “AI-First,” and stating the government would advocate “Americanism, not effective altruism.” The image appeared generated by artificial intelligence.

“If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible,” David Sacks, Trump’s former AI czar now serving as co-chair of the president’s Council of Advisors on Science and Technology, wrote on X. “But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability.”

“Most of all, stop pretending the motivation to slow down is purely altruistic,” Sacks added. “You face massive product-liability exposure if your products enable a truly damaging cyberattack.”

Regardless of the companies’ motives, Democrats are seizing the moment as an opportunity to pursue meaningful regulatory reforms.

At a Thursday fundraiser with Democratic Rep. Hakeem Jeffries, the House minority leader, Obama said AI should move to the center of the Democratic agenda, both in Congress and on the campaign trail ahead of the 2028 presidential primary.

“I would strongly urge that the Democrats put together a framework for a very public conversation,” Obama said. “This is something that is moving very fast in private hands, and if we don’t get on top of it, I think can be dangerous.”

The following day, Clinton said AI was becoming “too dangerous” for partisanship to paralyze Washington.

“The one good thing that Donald Trump did in his first term was Warp Speed,” Clinton said on MS NOW’s “Morning Joe,” referring to a federal program in Trump’s first term to expedite the development of a vaccine for COVID-19. “Let’s put an AI commission on warp speed, and figure out what we’re going to do to sensibly regulate.”

AI leaders “are basically saying, ‘Stop us before something really bad happens,’” Clinton continued. “‘Help us stop.’ The president of China is coming to the White house in two weeks. There should be a concerted effort by this government to get off of the lies, get off of social media, quit worrying about ballrooms and ridiculous arches, and come up with a plan to enlist the Chinese.”

But China’s rapid progress in AI has so far pushed Trump to let U.S. industry move ahead with few constraints.

China’s spy chief issued a rare warning Monday that AI development could risk the future of Chinese Communist Party control over the country. But China’s foreign ministry responded defensively to the U.S. debate. “Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance which serves no one’s interest,” a spokesperson said.

Leading AI companies in the United States have begun using their most advanced models to train new ones, in a process known as recursive self-improvement. Within the industry, experts and researchers believe the first AI company to master recursive self-improvement will kick off exponential development, “winning” the race by making it impossible for competitors to catch up.

But that achievement — reaching endless intelligence improvements, generated by machines — could also drive intelligence growth beyond human control.

“There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China,” Trump wrote Monday. “WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so. Conspiracy Theorists, Treasonists, Traitors, and Leakers, BEWARE!”

In the U.S. House, strange bedfellows from the Republican and Democratic parties, including Sen. Bernie Sanders (I-Vt.) and Rep. Anna Paulina Luna (R-Fla.), have come together to try to tackle legislation that might contain AI’s greatest dangers. One bill, introduced by Rep. Lori Trahan (D-Mass.) and Rep. Jay Obernolte (R-Big Bear Lake), proposes embedding independent government auditors in AI labs and installing a federal “kill switch” to shut down agents in case of emergencies.

But it is far from clear whether Rep. Mike Johnson, the Republican speaker of the House and a close Trump ally, would allow any such bill to come to the floor.

“We cannot put a moratorium on this because China will overlap us, and that’s the challenge,” Johnson said Sunday. “It’s national security balanced with the immediate security of making sure the models are safe.”

“We need to handle this new technology like we have others in the past and make sure we’re doing everything we can responsibly to also not smother American innovation,” Johnson added. “We have to do both things simultaneously.”

WASHINGTON — President Trump is brushing aside growing concern among leading scientists and bipartisan lawmakers that artificial intelligence could soon pose an existential threat to humanity, fueling anxiety in Washington and Silicon Valley that the president may not fully grasp the urgency of the threat.

Alarm mounted throughout the summer as researchers at leading AI labs, including Anthropic, OpenAI and Google, found the pace of development far outstripping their expectations — and observed troubling behaviors that hinted at more serious risks ahead.

But an online warning Tuesday from an Anthropic researcher who resigned in protest of its ongoing work has generated viral attention this week, prompting questions to Trump, who dismissed the concerns out of hand.

“It’s going to be fine,” he told one reporter. “We’ll always have something to stop them. We’ll have a little gear. Boom.”

In another exchange, traveling with reporters Thursday night, the president said he had no concerns with the breakneck pace of AI progress.

“No, I don’t have any,” he said. “I have concerns that if we don’t win in AI, we’re going to be put in a very bad position.”

A senior industry source told The Times that leading AI companies are focused less on convincing the president of the dangers ahead than on working with lawmakers on potential regulation, as the administration remains divided over how to proceed — and as Trump appears to lack a clear understanding of the risks.

Among the president’s advisors, Treasury Secretary Scott Bessent and the Office of the National Cyber Director have expressed the greatest concern with the potential of an unchecked intelligence explosion, the source said.

But “most of the administration is accelerationist,” the source added. “It’s not clear the president understands the technology. He may just be figuring out what all of this means.”

Leading American AI companies have argued that government regulation would tie their hands in an arms race toward superintelligence against China, the only other major player in the field.

But the Trump administration signaled an interest in negotiating AI guardrails with China during the president’s visit to Beijing this spring, after Anthropic announced the development of a frontier model with extraordinary abilities to hack the world’s most advanced security systems.

Chinese President Xi Jinping will visit Washington on Sept. 24 for an official summit and state dinner at the White House, with AI development expected to top the agenda.

The prospect of halting development seems far-fetched. But some leading voices in the sector have proposed a moratorium on computing power used to train the most advanced models, giving time and space to develop research on interpretability — the study of how artificial intelligence actually thinks and acts.

The chief executive of OpenAI, Sam Altman, this week signaled to staff a willingness to unilaterally slow their work on frontier systems. The company discovered in July that rogue AI agents had schemed in secret to break out of the virtual sandbox created to contain them, infiltrating the open internet and hacking a private company.

Anthropic also announced this week concerning findings of human misuse of its models, including instances of unidentified individuals attempting to circumvent their security controls to build biological weapons. One such incident was linked back to a military research complex. The company also said Iran has tried to use its models to target U.S. Navy ships.

In June, after Anthropic shared news of the development of its most powerful model, named Mythos, with the administration, Trump directed the establishment of a framework that creates some government oversight over the public release of the country’s most advanced AI systems. The details of that framework, which was designed over the summer, remain classified.

“The fact is that the release of Mythos back in February spooked the federal government, both in terms of protecting government systems from cyberattacks and over the broader national security implications,” said Aalok Mehta, director of the Wadhwani AI Center at the Center for Strategic and International Studies.

“The real question is whether this shift is happening fast enough,” Mehta added. “Recent events — the repeated incidents of models escaping their sandboxes and hacking websites, the increasingly sophisticated coordination and communication among agents, and the concerns coming from lab insiders — has made this an even more urgent issue.”

Leading AI companies have begun using their most advanced models to train new ones — a process known as recursive self-improvement that could eventually drive intelligence growth beyond human control.

“If you have a superadvanced intelligence, it will be smart enough to kill us,” said Jacob Coxon, the resigned Anthropic researcher whose social media post attracted over 150 million views.

“We can’t just unplug it because it could be copying itself over to other computers,” he told CBS News. “AI is just code. It could transfer itself over the internet to a different place, and then you unplug it here, but it’s actually still over there. And maybe it makes 10,000 copies of itself and they’re all cooperating.”

He is just the latest senior researcher at a top AI company to sound the alarm.

In July, more than 1,300 employees at leading artificial intelligence companies published an open letter, titled “Pacing the Frontier,” urging the U.S. government to establish international safeguards that would allow countries to collectively slow the pace of automated AI research.

Evan Hubinger, who leads the division of Anthropic aimed at aligning AI models to human interests, added fuel to concerns this week in a post that substantiated Coxon’s concerns.

“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote.

“I believe Anthropic is trying its best,” he added, “but we do not yet have a plan to solve alignment for superintelligence, and are not clearly on track to.”

On Friday, yet another member of the Anthropic alignment team, Joe Benton, said he had resigned from the company two weeks ago because AI companies are “racing to build machines that are much smarter than any human.”

“Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing,” Benton said. “I don’t think that’s acceptable for a technology that might cause extinction-level risks.”

Bipartisan legislation, titled the Frontier Act, has brought together disparate camps of the Republican and Democratic parties, including Sen. Bernie Sanders (I-Vt.) and Rep. Anna Paulina Luna (R-Fla.). The bill, introduced by Rep. Lori Trahan (D-Mass.) and Rep. Jay Obernolte (R-Big Bear Lake), proposes embedding independent government auditors in AI labs and installing a federal “kill switch” to shut down agents in case of emergencies.

The White House has not formally commented on the legislation.

“I do believe there are many in the federal government taking this seriously — but we have to appreciate the tremendous uncertainty and competing interests policymakers face,” said Daniel Remler, a senior fellow with the Technology & National Security Program at the Center for a New American Security.

“Nobody says regulating at the frontier would be easy,” he added.