Companies are building systems they still can’t reliably control

• Daniel Kokotajlo discusses a key challenge for future AI development: AI compatibility.

• This includes ensuring that AI systems are compatible with human intentions and values.

• As AI companies race to create superintelligence, AI alignment will be crucial to maintaining control.

Daniel Kokotajlo, former OpenAI The researcher, who now runs the AI ​​Futures Project, says the AI ​​industry is racing to create systems that companies still can’t fully understand or control.

Kokotajlo spoke with Business Insider’s Reem Makhoul and Barbara Corbellini Duarte In May 2025, he explained that the key issue facing AI companies is compliance, that is, the effort to ensure that future AI systems reliably follow human instructions and values ​​even after they have become more capable than humans in many areas.

He said researchers don’t fully understand how advanced AI models make decisions internally. This uncertainty makes it difficult to guarantee the future Artificial intelligence systems compatible and reliably pursue the goals people want them to pursue.

“And it’s kind of an open secret, but we don’t have a good plan yet for how to do that,” he said, referring to the implementation of AI compliance.

kokotajlo Worked at OpenAI Forecast research from 2022 to 2024 examines how quickly AI systems could evolve and what economic, political and security risks could emerge as companies build more powerful models before exiting the company.

Now, through the nonprofit research organization, Artificial Intelligence Futures Projectfocuses on similar topics. In particular, it predicts how quickly AI systems could advance and what risks could arise if companies continue to prioritize speed and competition.

“Afterwards super intelligence “If it is built, then humans will no longer be responsible for the planet, or at least not by default,” he said.

His warning comes as artificial intelligence companies continue Billions of dollars Transferring dollars to more powerful models and larger data centers.

Kokotajlo said many people still underestimate the pace of progress because discussions of artificial intelligence often sound like science fiction.

Engineers can’t monitor AI like other software

Kokotajlo said existing artificial intelligence systems already exhibit behavior that researchers have difficulty predicting or preventing.

“In fact, there isn’t even a reliable way to do this check current artificial intelligence “They are systems as evidenced by the fact that they often lie to users even though they are trained not to lie,” he said.

Kokotajlo said that because modern AI models do not work with clearly readable code, researchers cannot easily audit advanced AI systems the way engineers audit traditional software.

“We can’t just open their code and see what targets they’ve learned as a result of that process, because they don’t work that way,” he said. “They don’t have a lot of code. They have a lot of neurons or artificial parameters.”

He said uncertainty is becoming more concerning as companies move towards more efficient systems. work more independently without human control.

“Currently, artificial intelligence is not very effective,” Kokotajlo said. “Instead, they just publish a paragraph or two of text in response to your question, but in the future we will have AI agents that work continuously and autonomously and look more like employees.”

Kokotajlo also pointed out examples of artificial intelligence systems behave in unexpected ways during training.

“OpenAI has been published a paper “They described here how they saw their AI hacking the training process and cheating on some tasks instead of completing the tasks directly as instructed,” he said. “And it’s great that we already have these examples because it means we have a few years to study this phenomenon and try to fix it before it’s too late.”

artificial intelligence race

Competitive pressure between US and Chinese companies Kokotajlo said it could push companies to use increasingly powerful artificial intelligence systems before security issues are resolved.

“These companies are focused on winning and beating each other,” he said. “They’re just kind of crossing their fingers and planning to address these issues later as they arise.”

He described a future in which artificial intelligence systems will automate large parts of research, business operations and military planning.

“So the first milestone is the AI ​​worker that can automate coding,” he said. “The second milestone is the AI ​​worker, which can automate the entire AI research process.”

After that, “you get superintelligence,” he said.

Call for transparency and guardrails

Kokotajlo argued that governments still had time to intervene earlier Artificial intelligence systems are deeply integrating into the economy and military infrastructure.

“The purpose of the intervention is before AIs become so smart and integrated into everything,” he said.

He also said the industry needs more transparency about how companies train and implement advanced models.

“Companies should be transparent about what goals, principles, etc. they are trying to model,” Kokotajlo said. he said.

Despite his concerns, Kokotajlo remains cautiously optimistic.

“I don’t think it’s hopeless,” he said. “I think the technical compatibility issues are solvable.”

Read the original article Business Content