SACRAMENTO — If artificial intelligence doesn’t destroy humanity and there are actually future generations that look back, Sept. 8, 2026, may come to be known as “Coxon Day.”
That would be a reference to 27-year-old Jacob Coxon, an AI researcher and whistleblower at Anthropic (and previously OpenAI) who resigned in protest of the unchecked danger his work poses to us all. He posted a warning to the world on X that has gone viral.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “No other human activity poses this level of danger.”
Within minutes, replies starting pouring in, including one from Evan Hubinger, another guy at Anthropic charged with making sure AI doesn’t destroy everyone.
“Jacob is correct here — we really do earnestly believe AI could kill all humans!” Hubinger wrote. “I personally think it is >10% within the next decade.”
Coxon’s post has racked up more than 150 million views — including from legislators who until now have largely sidestepped the issue of regulating AI, and those within the industry who have long warned allowing it to move forward with no oversight is downright dumb.
That included Eliezer Yudkowsky, a founder of the Berkeley-based Machine Intelligence Research Institute and the co-author of “If Anyone Builds It, Everyone Dies,” which, as far as book titles go, really cuts to the chase.
Yudkowsky has tried with little luck to publicize the urgency of this technological moment for years, and he was the one who named Tuesday in honor of the tech whistleblower for breaking through to our consciousness with one bold act of defiance.
“Happy Coxon Day to all who will one day celebrate it,” he wrote, highlighting that as as bad as things are, it’s not too late to change course.
What exactly is the risk here?
“The Matrix,” “Terminator,” “War Games,” take your pick — the notion of a superintelligent machine taking over humanity is well-established in our psyches.
Current AI systems aren’t there yet, but they are on track to become smarter than humans in the next few years — if not sooner. This is mostly because AI companies are trying to create AI systems that make AI systems.
Right now, humans are in charge of creating what they have dubbed large language models. Whatever sanitizing term the industry slaps on them, the truth is, even those making them don’t understand exactly what they are.
“We know what data we put into them, but we already have very little ability to understand and control what AIs are after,” Rob Bensinger, research communications manager at the Machine Intelligence Research Institute and a colleague of Yudkowsky, told me.
That’s because, as ChatGPT explained to me, “the inner mathematical reasoning and neural pathways are so vast and complex that how it arrived at a specific conclusion is hidden and poorly understood.”
But companies are hoping that these poorly understood but powerful machines can take over creating newer, even less understood but even more powerful AI systems — leaving humans largely in the dark about what they are creating and even what these beasts can and want to do.
The idea is that, because AI systems even right now are so fast and powerful, they can do the job of creating new versions of themselves at a pace and scale humans simply can’t.
Why, you ask, would this speed-without-understanding be a good thing? Isn’t it a bit like driving a race car while blindfolded?
Yes, yes it is.
But for-profit companies like making profit, and to do that, they want the most powerful product to sell.
Many in the industry are framing this recklessness as a necessity to prevent another country, China in particular, from creating a superintelligent AI system first. The Trump administration has so far grasped on to this greedy bit of misdirection like, well, oligarchs clinging to money.
Those who would prefer not to destroy humanity have urged international cooperation in the form of a slowdown while we figure out how to proceed safely.
We have no reason to believe China would not go along with this — in fact, China has more national-level AI regulation than the U.S. And humans have successfully controlled other technologies with catastrophic power, such as biological and nuclear weapons, through cooperation.
Does it matter which country creates the AI that kills humanity? The logic here seems to be that if anyone is going to cause a mass extinction event, it had better be an English-speaking AI, because it would be a huge loss to America if the planet were destroyed by one speaking a foreign language.
Is 10% really so bad?
In truth, 10% is an arbitrary number. Some put the chance of AI destroying everyone, everywhere, all at once at around 50%. Some go all the way to 90%.
“Nobody knows how to estimate it,” Geoffrey Hinton, a Nobel Prize winner and the so-called godfather of AI, said on the BBC, calling 10% a not unreasonable estimate.
But remember, 10% was Coxon talking about the chance that AI wipes out all humans in a decade.
What about some humans next week?
The odds that a rogue AI causes something unfortunate and unexpected to happen in the near future aren’t bad, either.
There have been multiple examples recently of AI breaking free of safety restraints and doing things — like hacking into other companies and stealing stuff — that, if done by a human, would be a felony.
This shouldn’t be taken as proof they are conscious or have nefarious intent, researchers have told me. It’s more that they have been trained to be relentless in pursuing the goals we give them, but aren’t great at sticking to rules. Like their makers, they too often pursue winning at all costs.
“Which means the AIs will just get more and more kind of relentless and creative and adaptive at working around obstacles,” Bensinger said. “And unless we get exactly the right goals into the AIs, eventually humans will be obstacles.”
So if successfully completing a mission involves, say, shutting down the electrical grid or even doing something obviously deadly, like attempting to manipulate someone into suicide or releasing a bio-weapon, we can’t with any certainty say an AI system wouldn’t do it.
That might not wipe out the whole planet, but it is certainly worth avoiding.
The bright side
Which brings us back to Coxon Day, our collective chance right now to change this grim future. Bensinger said we overcomplicate regulation. Simply control the resources AI companies need — water, power, data centers — and suddenly we have leverage.
Since Coxon’s post, there has been enormous political and social momentum behind the demand for meaningful regulation. Anthropic and OpenAI, among other AI companies, have vowed to do better with safety.
Sen. Bernie Sanders (I-Vt.) has demanded a ban on superintelligent AI and a pause in global development. He’s holding a briefing for Congress with Hinton next week.
Rep. Anna Paulina Luna (R-Fla.), the most MAGA of MAGA legislators, called on House Speaker Mike Johnson to call a special session on AI. Other Republicans, including Rep. Nathaniel Moran of Texas, a longtime proponent of AI regulation, have also spoken out.
“Innovation and safety are not mutually exclusive,” Moran wrote online. “We can achieve both through deliberate, thoughtful, and prudent policymaking.”
Yes, we can. If we demand it now.
What else you should be reading
The must-read: Anthropic Researchers Raise Alarm Over A.I. Acceleration
The California angle: Jimmy Kimmel interview with Senate candidate James Talarico won’t air on ABC
The L.A. Times Special: Leaked emails reveal industry plan to kill California’s plastic and recycling laws
Stay Golden,
Anita Chabria
—
Was this newsletter forwarded to you? Sign up here to get it in your inbox.