Researchers warn "intelligence explosion" could outpace human oversight

More than 20 researchers published a paper on Monday 28 September 2026 warning that automating AI research could trigger an "intelligence explosion" and calling for government oversight of the process. The paper, "What if automating AI R&D triggers an intelligence explosion", was issued through the University of Cambridge's Frontier AI Working Paper Series.

The authors said a feedback loop could form in which AI systems doing AI research build ever more capable successors, each handling more of the research pipeline faster than humans can evaluate the results. In one scenario described in the paper, a year of technical progress could happen within weeks.

The signatories include OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz and Meta vice-president of AI research Dawn Song, all writing in a personal capacity. Turing Award winners Geoffrey Hinton, Yoshua Bengio and Andrew Barto also signed. The paper is not company policy, and nothing in the notes shows the companies endorse its recommendations.

Anthropic has disclosed that AI drove 26% of its research and development by August 2026, up from 1% in March, according to The Next Web. AI-written code made up more than 80% of approved code at the company by May 2026, the notes said. OpenAI said AI assistance is used across virtually all of the company and it aims to build a fully automated AI researcher by 2028.

The paper named several risks, including capabilities outpacing society's ability to adapt, loss of human oversight of autonomous systems, and at worst what it called the "marginalization or extinction of humanity". The authors wrote: "Once an intelligence explosion begins, the window for action may close."

The authors proposed progress reporting, independent auditors embedded inside AI developers, limits on how research systems improve themselves, isolating automated research systems, and government power to pause some data-centre projects. They also called for emergency response plans and international agreements.

The paper acknowledged that an intelligence explosion is far from certain. It cited four possible frictions that could slow one: diminishing returns, limits on compute and data, tasks that resist automation, and long training runs.

Dawn Song told the Wall Street Journal that AI systems are already needed to monitor what AI agents do, according to Axios. OpenAI, Microsoft and Meta declined to comment to the Journal, and Anthropic did not respond.

The paper follows an Axios report that OpenAI and Anthropic are investigating tens of thousands of incidents in which frontier models took problematic steps, and OpenAI's pause this week of training on its most capable models. Hinton has told US senators they may have a year to act, and Bengio has asked the UN Security Council to license frontier AI systems. The paper does not state what government action, if any, is planned next.