Fears of an AI breakthrough force the U.S. and China to talk

CHONGQING, China — Three years ago, in the idyllic town of Woodside south of San Francisco, the United States and China held their first high-level talks on the dangers posed by artificial intelligence. President Xi Jinping and his longtime foreign minister appeared serious in their belief that a channel — a red phone for AI in emergencies — should be established between Beijing and Washington.

They authorized a diplomatic effort to begin in Switzerland in 2024, just months before the US presidential election. A large U.S. delegation arrived with high hopes, but those hopes were suddenly dashed, according to four sources who participated in the talks. The Chinese contingent dismissed America’s concerns about AI getting out of control as academic and almost theoretical, quickly turning the topic to export controls seen in Beijing as yet another US effort to hold China back.

“Naturally, they view any American diplomatic initiative that involves limiting or restricting a capability in one way or another as a trap,” Jake Sullivan, President Biden’s U.S. national security adviser, said in an interview.

Despite the mistrust and the Democrats losing the White House to Donald Trump, a deal was struck in Peru in November of that year in which both sides agreed to keep AI out of command and control of nuclear weapons.

“This was the breaking of the seal that we could actually do something about AI,” Sullivan said. “During the transition, I told the incoming Trump team that they really needed to pursue this dialogue. But the Trump administration’s view was much more laissez-faire, and they weren’t very interested in that.”

“Everything has changed in the last few weeks,” he added.

Once eager to weaponize for technological superiority, the Trump administration is now calculating for the first time the power that artificial intelligence could unleash if left unchecked.

In a surprise reversal, quiet talks were taking place on the emergency channel to discuss reviving talks ahead of President Trump’s state visit to China this week, officials told The Times, in a surprise reversal as joint alarm was stirred in Beijing and Washington over the launch of Anthropic’s powerful new model, the Mythos.

The White House is trying to create a communications channel for artificial intelligence like others it has “in many areas of intense focus with the United States and China,” a senior administration official told reporters Sunday.

“I think the formality and what that communication channel looks like has not yet been determined,” the official said. “But we want to use this opportunity with the leaders’ meeting to start a conversation. We have to establish a communication channel on this issue.”

Mythos’ abilities are viewed throughout industry and government as the same as: An unprecedented cyber weaponcan infiltrate and exploit digital communications systems, including government databases, financial institutions, and healthcare programs, with untold consequences.

It’s unclear whether an announcement will happen this week. Any talks on artificial intelligence regulations between the United States and China to devise some kind of arms control agreement governing the use of a technology neither side fully understands or controls would be fraught with doubt, misunderstanding and risk, experts say.

“There is currently virtually no support from U.S. policymakers to engage in formal discussions with China on AI governance,” said Aalok Mehta, director of the Wadhwani Center for Artificial Intelligence at the Center for Strategic and International Studies.

“The logic is that this is a winner-take-all race, and it is imperative to accelerate the advancement of artificial intelligence to ensure the United States wins this race,” Mehta said.

America is ahead

China will enter these discussions with a strong argument that US leadership in AI and the dominant strategy of American AI companies is pushing the world towards an alarming limit.

Every major US player in the arena (OpenAI, Google, Anthropic, Microsoft, and Meta Platforms) is competing to be the first to build a model capable of artificial general intelligence, or AGI, a threshold for which there is no common definition, but most agree that performing any intellectual human task will require a model.

The prevailing theory is that whoever reaches AGI first will reap an exponential reward: self-education, improving intelligence over and over again, growing exponentially, and leaving all competitors behind.

Chinese companies, by contrast, are pursuing a state-approved strategy that focuses on integrating AI into individual sectors and systems, training models to improve individual tasks and accelerate growth through a more tailored approach.

“The Chinese believe there is not just one race, but multiple races,” said Scott Kennedy, a senior advisor on Chinese business and economics at the Center for Strategic and International Studies. “While the US is focused on achieving AGI, China is focusing on the diffusion and application of AI to the rest of their economy (manufacturing, humanoid robots, all aspects of the internet of things).”

Chinese academics, AI industry insiders and successive administrations have questioned Beijing’s strategic thinking and candor.

“It’s been so baked into society here that AGI is going to have this transformative potential that people can’t believe China isn’t focused on this as well,” said Matt Sheehan, an expert on global technology issues focused on China at the Carnegie Endowment for International Peace. “He says he’s focused on applications, but is this a fake AGI program hidden somewhere in the mountains?”

But many insiders believe Beijing’s guidance to Chinese companies reveals its true intentions.

“They’re not as rife with AI as the United States, and I think that’s still the case today,” Sullivan said. “So they viewed a lot of the talk in the US about extreme border risk — misalignment and loss of control — as somewhat abstract and not very relevant to how they see AI spreading in China.”

Although China’s progress has exceeded U.S. expectations—especially since DeepSeek launched its model a year ago—the government has focused its computing power on specific applications rather than the broad strategy needed to develop more powerful models that could move toward AGI.

“It’s not just chips. It’s money, too,” Sheehan added. “China’s leading companies are much more financially constrained than U.S. companies. There are concerns about a bubble here, but OpenAI is valued at close to $800 billion. Leading Chinese companies going public are valued at $20 billion. There’s just a gap in available funding.”

Still, some in the U.S. government fear that China would not need similar computing power if it stole the technology wholesale.

This is not simple to do. But last month, the White House Office of Science and Technology Policy accused Chinese actors in a memo of “engaging in industrial-scale campaigns to distill U.S. border AI systems,” essentially replicating the performance of existing state-of-the-art models “at a fraction of the cost.” The statement does not accuse Beijing of approving this activity.

A note was added that in this process, carefully established security protocols were deliberately eliminated.

China’s negotiating advantage

Regardless of its strategic calculation, China will begin negotiations with the Trump administration, which is following the race, even if they cannot agree on the nature of the finish line.

Theoretically, AGI can reach a stage of iterative self-improvement that results in the loss of human understanding or control. But if it is only the Americans and not the Chinese who are trying to reach this threshold, then who is responsible for stopping it?

Daniel Remler, who directed artificial intelligence policy at the State Department during the Biden administration and took part in the Geneva talks, cast doubt on China’s lack of interest in AGI and its claims that its risks are being ignored. He said China’s lagging behind in the race was not a strategic design.

“Chinese technologists are close observers of the US AI ecosystem and sometimes say what they think,” Remler said. “Many people were affected [Mythos] model to the point of despair. Leaders at China’s top AI labs have been vocal in recent months, even before Mythos, about how limited they are in terms of frontier computing. “Some said they would never be able to catch up with their American rivals.”

Talks at this point in the race could follow a familiar pattern in the recent history of U.S.-China diplomacy, in which Beijing argues that the United States is behind in development, resulting in a handicap at the negotiating table and greater concessions.

In other competitive areas, such as China’s entry into the World Trade Organization and cybersecurity negotiations between Beijing and the Obama administration, agreements were ultimately reached that Washington believes, in hindsight, put American companies at a disadvantage.

Remler added that the Trump administration “must approach AI diplomacy with China with clear expectations tied to our own national interests.”

Silicon Valley is divided on the regulation of artificial intelligence. Built on concerns that other AI companies were not taking security and compliance concerns seriously, Anthropic sounded the alarm to the Trump administration about its model, Mythos, prompting consideration in the White House about the best path forward.

After meeting with leaders of America’s major banks about the vulnerabilities, a startled Treasury Secretary Scott Bessent recommended internal reviews of future model releases to the US government; this is already an ongoing practice in China, where training parameters for models, known as “weights,” are made public.

Even the suggestion of government oversight sparked a backlash in Silicon Valley. Last week, the White House sent a memo to industry allies to reassure them that submitting new models to federal review would be strictly voluntary.

Experts believe that if talks on artificial intelligence resume between Washington and Beijing, the negotiations will be much more complex than those that resulted in arms control agreements on nuclear weapons in the Cold War.

The superpowers will be discussing not only threats of destabilization to the global financial system, but also fears of nuclear proliferation – the falling of advanced AI tools into the hands of bad actors interested in using biological or cyber weapons that could target both countries.

And ultimately, they would have to decide whether to discuss regulating the integration of artificial intelligence into the Chinese and U.S. militaries, an almost elusive goal among the world’s greatest adversaries where trust is lowest and hardest to verify.

Those in the industry who fear most what artificial superintelligence might bring have told the Trump administration that talks with China are an existential necessity.

Dario Amodei, CEO and co-founder of Anthropic, speaks at an event in New York in 2025.

(Michael M. Santiago/Getty Images)

But even within Anthropic, which advocates diplomacy, there are concerns that Beijing could exploit its current disadvantage to propel American industry to the pinnacle of its greatest success.

Rather than pushing for a single comprehensive agreement, industry insiders have recommended the administration make targeted agreements to reduce specific risks, such as the nuclear command and control pact with Beijing, two industry sources said.

In private conversations, both Xi and Chinese Foreign Minister Wang Yi appeared to understand that the seriousness of the technology evolving before them required some form of cooperation, Sullivan said.

“On a conceptual level, I believe they had an opinion on this and allowed it,” Sullivan said, “but I believe their level of urgency was much lower than ours, and they viewed this as a longer-term process that would play out over time.”

“Their level of urgency and stake in this has increased,” he added.