Fears of an AI breakthrough force the U.S. and China to talk
CHONGQING, China — Three years ago, in the idyllic town of Woodside south of San Francisco, the United States and China held their first high-level talks on the dangers posed by artificial intelligence. President Xi Jinping and his longtime foreign minister appeared serious in their conviction that a channel should be a established between Beijing and Washington — a red phone for AI in case of emergencies.
They authorized a diplomatic effort that would begin in 2024 in Switzerland, only months before the U.S. presidential election. A large U.S. delegation arrived with high hopes that were abruptly dashed, according to four sources who attended the talks. The Chinese contingent dismissed American concerns over runaway AI as academic, almost theoretical, quickly turning the conversation to export controls seen in Beijing as yet another U.S. effort to hold China back.
“They naturally view any American diplomatic initiative involving limitations or restrictions of one flavor or another on a capability as being a trap,” Jake Sullivan, U.S. national security advisor under President Biden, said in an interview.
Despite the distrust — and Democrats losing the White House to Donald Trump — an accord was struck in November of that year in Peru, where both sides agreed to keep AI out of the command and control of nuclear weapons.
“It was a breaking of the seal that we could actually do something on AI,” Sullivan said. “In the transition, I told the incoming Trump team that they should really pick up that dialogue. But the Trump administration’s view was just far more laissez-faire, and they didn’t seem particularly interested in it.”
“That’s all changed in the past few weeks,” he added.
A Trump administration once eager to gun for technological supremacy is now, for the first time, reckoning with the power AI could unleash if left unchecked.
In a surprise reversal, quiet discussions have taken place ahead of President Trump’s state visit to China this week to explore reviving talks on an emergency channel, officials told The Times, prompted by shared alarm in Beijing and Washington over the debut of Mythos, Anthropic’s powerful new model.
One senior administration official told reporters Sunday that the White House was looking to create a channel of communication for AI like others that they have “in many areas that have intense focus with the U.S. and China.”
“I think what that channel of communication looks like, its formality and what that looks like, is yet to be determined,” the official said, “but we want to take this opportunity with the leaders meeting to open up a conversation. We should establish a channel of communication on that matter.”
Mythos’ capabilities are seen across the industry and government as those of an unprecedented cyberweapon, able to infiltrate and exploit digital communication systems — including government databases, financial institutions and healthcare programs — with untold consequences.
Whether an announcement will come to fruition this week is not yet clear. Any talks between the United States and China over AI regulations — designing some kind of arms control agreement governing the use of a technology that neither side fully understands or controls — will be fraught with suspicion, misunderstandings and risk, experts say.
“Right now, there is almost no support from U.S. policymakers to engage in formal discussions on AI governance with China,” said Aalok Mehta, director of the Wadhwani AI Center at the Center for Strategic and International Studies.
“The logic is that this is a winner-takes-all race,” Mehta said, “and that it’s imperative to accelerate AI progress to ensure that the United States wins that race.”
America in the lead
China would enter those discussions with a powerful argument, that U.S. leadership in AI — and the prevailing strategy of American AI companies — is propelling the world to a fraught frontier.
Every major U.S. player in the arena — OpenAI, Google, Anthropic, Microsoft and Meta Platforms — is racing to be the first to build a model capable of artificial general intelligence, or AGI, a threshold without a common definition, but that most agree will require a model to perform any intellectual human task.
The prevailing theory is that the first to achieve AGI will secure a prize that multiplies itself: a self-training, recursively improving intelligence, growing exponentially and leaving all competitors in its wake.
Chinese companies, by contrast, are following a state-sanctioned strategy focused on integrating AI into siloed industries and systems, training models to improve individual tasks and accelerate growth in a more tailored approach.
“The Chinese believe there is no single race, but multiple races,” said Scott Kennedy, senior advisor on Chinese business and economics at the Center for Strategic & International Studies. “The U.S. is focused on achieving AGI, while China is focused on diffusion and applications of AI into the rest of their economy — manufacturing, humanoid robotics, all aspects of the internet of things.”
China scholars, AI industry insiders and successive administrations have questioned Beijing’s strategic thinking and forthrightness.
“It’s so baked into the community here that AGI will have this transformative potential that people can’t believe China isn’t focused on this, as well,” said Matt Sheehan, a scholar of global technology issues at the Carnegie Endowment for International Peace with a focus on China. “It says it’s focused on applications, but is that a fake out for an AGI program hidden in the mountains somewhere?”
But most insiders believe that Beijing’s guidance to Chinese companies reveals its true intentions.
“They are not as AGI-pilled as the United States is, and I think that remains the case today,” Sullivan said, “so they regarded a lot of the conversation in the U.S. around extreme frontier risk — misalignment and loss of control — as a bit abstract, and not really as relevant to how they saw AI diffusing in China.”
President Biden greets Chinese President Xi Jinping in Woodside, Calif., in 2023.
(Doug Mills / Pool Photo)
Although China’s progress has exceeded U.S. expectations — especially since DeepSeek released its model over a year ago — the state has focused computer power on specific applications rather than the broad strategy needed to develop more powerful models capable of advancing toward AGI.
“It’s not just chips. It’s money,” Sheehan added. “China’s leading companies are much more financially constrained than U.S. companies. There’s concern over a bubble here, but OpenAI is valued at something near $800 billion. Leading Chinese companies that have gone public are valued at $20 billion. There’s just an orders-of-magnitude gap in available financing.”
Still, some in the U.S. government fear China won’t need comparable computing power if it simply steals the technology wholesale.
Doing so isn’t simple. But last month, in a memo, the White House Office of Science and Technology Policy accused Chinese actors of “industrial-scale campaigns to distill U.S. frontier AI systems,” in effect replicating the performance of the most advanced existing models “at a fraction of the cost.” The memo did not accuse Beijing of endorsing the activity.
In the process, the memo added, carefully constructed security protocols are deliberately stripped away.
China’s negotiating advantage
Whatever its strategic calculus may be, China would enter talks with the Trump administration trailing in the race — while disagreeing on the nature of the finish line.
AGI, in theory, could reach a stage of recursive self-improvement that results in a loss of human understanding or control. But if it is only the Americans, and not the Chinese, seeking to reach that threshold, then who is responsible to stop it?
Daniel Remler, who led AI policy at the State Department during the Biden administration and took part in the Geneva talks, cast doubt on Chinese claims of disinterest in AGI and ignorance of its risks. China falling behind in the race is no strategic design, he said.
“Chinese technologists are close observers of the U.S. AI ecosystem, and sometimes they say what they think,” Remler said. “Many were impressed by the [Mythos] model to the point of despair. Leaders in China’s top AI labs have been vocal in recent months, even before Mythos, about how compute-constrained they are at the frontier. Some have said they may never catch their American competitors.”
Talks at this point in the race could follow a familiar pattern in the recent history of U.S.-China diplomacy, in which Beijing claims it is behind the United States in development, ultimately securing a handicap and greater concessions at the negotiating table.
In other competitive domains — such as with China’s entry into the World Trade Organization and in cybersecurity negotiations between Beijing and the Obama administration — agreements were ultimately reached that Washington believes in hindsight disadvantaged American companies.
The Trump administration, Remler added, “needs to approach AI diplomacy with China with clear-eyed expectations anchored to our own national interests.”
Silicon Valley itself is divided over regulating AI. Anthropic, which was founded on concerns that other AI companies were failing to take safety and alignment concerns seriously, raised alarms over Mythos, its own model, to the Trump administration, a moment that has prompted reflection at the White House on the best path forward.
Spooked after meeting with leaders from America’s top banks over their vulnerabilities, Treasury Secretary Scott Bessent internally advised U.S. government reviews of future model releases — a practice already underway in China, where the training parameters for models, known as “weights,” have been publicly released.
Even the suggestion of government oversight sparked backlash from Silicon Valley. Last week, the White House sent out a memo to reassure industry allies that submitting new models for federal review would be strictly voluntary.
If talks ultimately resume between Washington and Beijing on AI, experts believe the negotiations would be far more complex than those that resulted in arms control agreements governing nuclear weapons in the Cold War.
The superpowers would not only be discussing threats of instability to the global financial system, but also fears of proliferation — advanced AI tools getting into the hands of bad actors interested in using bio- or cyberweapons that could target both countries.
And they ultimately would have to decide whether to discuss regulating the integration of AI into the Chinese and U.S. militaries, an almost unfathomable goal between the world’s biggest adversaries, where trust is lowest and verification would be hardest.
Those in the industry who most fear what artificial superintelligence could bring have told the Trump administration that talks with China are an existential necessity.
Dario Amodei, the chief executive and co-founder of Anthropic, speaks at an event in New York in 2025.
(Michael M. Santiago / Getty Images)
But even within Anthropic, which has championed diplomacy, there are concerns that Beijing could exploit its current disadvantage to entangle American industry at the cusp of its crowning achievement.
Rather than pushing for a single sweeping agreement, industry insiders are advising the administration to pursue targeted deals with Beijing to mitigate specific risks, like the pact on nuclear command and control, two industry sources said.
In private, both Xi and Chinese Foreign Minister Wang Yi seemed to understand that the gravity of the emerging technology before them required some form of cooperation, Sullivan said.
“At a conceptual level, I believe they had a conviction on that and authorized it,” Sullivan said, “but I believe their level of urgency was considerably lower than ours, and saw this as a longer-term process that would play out over time.”
“Their level of urgency and their stake in it has gone up,” he added.
