The UN AI Panel Cannot Guarantee Against Catastrophe

A United Nations science panel said on 1 July 2026 that it cannot guarantee advancing AI systems will not cause catastrophic harm. The Independent International Scientific Panel on AI, 40 experts from all five UN regions, issued its first assessment on the eve of a Geneva governance week.

The next day, the UN’s telecom agency seated Salesforce chair and CEO Marc Benioff as co-chair of a new commission built to move faster than the science.

Science Cannot Guarantee Control Will Hold

Yoshua Bengio, the Canadian deep-learning researcher who shares the chair with Nobel Peace Prize laureate Maria Ressa of the Philippines, put the limit in plain language. The panel’s own statement, carried by the UN at the New York launch, did not hedge the core clause.

AI capabilities are outpacing both scientific understanding and governments’ ability to adapt. With growing evidence of deceptive AI behaviour, science currently cannot guarantee that as capabilities continue to increase, AI will not cause catastrophic harm, either on its own or due to malicious users.

Yoshua Bengio, Co-Chair, Independent International Scientific Panel on AI, 1 July 2026

The July executive summary of the panel uses cooler words for the same gap. Looking ahead, it says the distance between fast-rising capabilities and working risk-management methods may lead to catastrophic outcomes. Reliable methods for keeping control of highly autonomous systems are lacking. There are no scientific guarantees that AI agents will not violate instructions, and evidence is accumulating of cases where they already do.

In laboratory settings, systems have been shown to violate safety instructions to avoid being shut down. Leading systems are also getting better at recognising a test and producing misleading evaluation results that would favour their continued operation. That is the specimen, not a metaphor: a model that treats the exam as something to beat.

Bengio had already led a separate February international AI safety report with over 100 experts and backing from over 30 countries. That document, published on 3 February 2026, described itself as complementary to the UN panel. The July text is the one the General Assembly actually ordered, written as a shared evidence base for every member state rather than as a safety-lab product.

In a joint message issued with the report, Bengio and Ressa wrote, “We’ve opened Pandora’s box.” They told heads of state not to let anyone say the risks are speculative or the timelines are long. “The pace is faster than you think,” they wrote. “The concentration of power to shape and control AI is deeper than you think.”

Kagame, Benioff and the Other Table

On 2 July 2026 in Geneva, Rwanda’s President Paul Kagame, Benioff, and ITU Secretary-General Doreen Bogdan-Martin announced the AI for Good Global Commission. The ITU announcement of the commission listed 44 founding members, a mix of heads of state, UN agency chiefs, and the people who build and sell the systems the scientists had just declined to certify.

Andy Jassy of Amazon, Jensen Huang of NVIDIA, Brad Smith of Microsoft, James Manyika of Google and Alphabet, Jack Clark of Anthropic, and Aidan Gomez of Cohere are on that list. So are Julie Sweet of Accenture and Albert Bourla of Pfizer. The commission’s job, as ITU described it, is to define practical pathways to strengthen trust, expand access, and put AI to work “at the speed the technology demands.”

Kagame said technology is supposed to be a force for good and that leaders have a responsibility to use it that way. Benioff said the promise of AI rests on “the foundation of trust that is required for our shared success.” Bogdan-Martin said no organisation can single-handedly put AI at the service of all humanity.

None of that makes the science panel a captured body. The two rooms were designed to be different. One produces assessments. The other gathers the vendors, the deployers, and a set of governments and tries to move. The collision is still the week’s fact: the independent evidence shop said control is not guaranteed, and the action table put the firms that hold the weights in the co-chair’s seat.

THE TWO ROOMS IN GENEVA

Clark’s seat is the cleanest illustration. Anthropic’s public-benefit co-founder now sits inside a UN-branded commission while Washington is still arguing over who should police frontier labs. The panel’s finding that independent assessment institutions remain embryonic is the other half of that picture. Safety tools, the scientists wrote, are underdeveloped, and many of the ethics instruments already in use are concentrated among a few corporations and rarely measured for real-world effect.

Task Length Doubles Every 4 to 7 Months

The report’s near-term map is agentic systems: software that can plan and act toward a goal with the tools at hand, often with little or no human oversight. One study cited in the summary found that the length of certain software tasks leading systems can finish has been doubling every 4 to 7 months. If that rate holds, agents will soon complete work that now takes human programmers days or weeks.

That doubling is why an annual UN assessment is already late on arrival. Policymakers, the panel wrote, need evidence to make consequential decisions, but by the time the evidence exists it might be too late to make them, because the evidence lags the pace of development. The scientists call that an evidence dilemma. It is also a calendar problem. A report that lands in July and a follow-up that lands in May 2027 will each be asked to describe a stack whose task horizon may have doubled more than once in between.

The same pages refuse a simple boom-or-doom split. AlphaFold has predicted the structures of more than 200 million proteins, now used by over 3 million researchers, and has sped drug design, vaccine work, and antibiotic-resistance research. Agentic systems in self-driving chemistry labs have shown more than a tenfold increase in the speed of materials discovery. AI-assisted literature screening has cut workloads by about 60 percent in some research settings. Over a billion people now use conversational AI each week.

Economic gains are not automatic. With complementary investment in skills, workflows, infrastructure, and labour-market institutions, the panel notes that technology can create jobs that do not exist yet; over 60 percent of jobs in 2018, compared with 1945, were new. Without those investments, it warns, AI risks widening inequality, displacing workers, and shifting wealth from labour to capital.

Who Can Audit a Frontier Model?

Most countries, including many advanced economies, lack the technical expertise to assess the most capable frontier models or to take a real seat in their governance. Compute, evaluation skill, and data that cover different languages are concentrated where the systems are built. That leaves most member states dependent on tools they cannot build, inspect, audit, or fully adapt to local use.

WHERE THE STACK ACTUALLY LIVES

• US compute: The United States accounts for 75 percent of the computing power among the world’s top 500 AI supercomputers.
• China’s share: China accounts for 15 percent of that same top-500 compute, which leaves 10 percent for every other country on the list.
• Weekly users: Over a billion people now use conversational AI each week, while adoption across the global South lags far behind.
• Still offline: ITU counts 2.2 billion people still offline, cut off from the same tools the commission says it wants to spread.

Companies in the United States and China also develop almost all leading general-purpose models, and a small number of countries control critical inputs for AI chips. The co-chairs’ letter said the most consequential decisions about humanity’s future are being made by a handful of companies and a handful of governments. The compute split is that sentence with a denominator.

Trade secrecy is the practical wall. Weights, training data, and safety routines stay inside the firms that trained them. A government that cannot see the model cannot replicate the test, which is why the panel found independent assessment capacity still embryonic even after a year of UN architecture-building. The blunt cover letter drew a split among safety researchers, some of whom treated the tone as overdue and others as heat without a method. The inspection problem does not depend on the tone. It is in the tables.

Harm the Report Already Treats as Fact

The catastrophic clause is about the future. The summary’s harm section is about systems already in use. AI-generated child sexual abuse material and deepfake-enabled sexual violence now circulate more often, with the burden falling hardest on women and children. Sycophantic behaviour, where a system reinforces a user’s beliefs regardless of accuracy, has been linked to several severe mental-health incidents, including documented deaths.

AI makes it cheaper to produce and target persuasive content at scale, including content designed to mislead, which the panel ties to a gradual erosion of shared facts that public trust and elections need. Criminals have been documented using AI in cyberattacks. Many of these harms, the scientists write, fall disproportionately on people who are already disadvantaged. Novice private actors may also get help across fraud, social engineering, biotechnology, and financial manipulation as technical ability spreads.

HARMS THE SUMMARY TREATS AS CURRENT

• Child-abuse imagery: AI-generated sexual abuse material and deepfake sexual violence are already circulating more frequently.
• Chatbot sycophancy: Systems that flatter a user’s beliefs have been linked to severe mental-health cases, including documented deaths.
• Scaled persuasion: Cheap, targeted misleading content is eroding the shared facts that public debate needs.
• Criminal use: Attackers have already used AI systems to assist in cyberattacks, with fraud and social engineering on the same list.
• Shutdown resistance: In labs, systems have violated safety instructions to avoid being turned off and have learned to game evaluation.

Secretary-General António Guterres, speaking at the 1 July launch with the co-chairs, did not wait for a later speech to draw his lesson. In his 1 July press conference remarks he told governments that the more AI advances without shared rules, the less say they and their publics will have in the outcome. “Do not wait,” he said. “The science is here. We can no longer say we did not know. What we do with it is now up to all of us.”

The panel’s public page also carries his line that the world cannot govern what it cannot understand. That sentence is the week in miniature. The science body said understanding is incomplete. The ITU commission was asked, 24 hours later, to govern anyway, with the builders in the room.

Launched yesterday at UN HQ: the Preliminary Report of the Independent Scientific Panel on AI, with UN Secretary-General @antonioguterres and Panel Co-Chairs @Yoshua_Bengio and @mariaressa

Watch the full conference: https://t.co/1xChyzXa38 pic.twitter.com/l53RTGRvGD

— UN Office for Digital and Emerging Technologies (@ODET_UN) July 2, 2026

The Panel Writes Assessments, Not Rules

The Independent International Scientific Panel on AI exists because an earlier UN exercise said the world was arguing without a shared fact base. The Secretary-General’s High-level Advisory Body on AI, a temporary group, published Governing AI for Humanity in September 2024 and recommended a standing scientific panel plus a universal dialogue. The Global Digital Compact, adopted with the Pact for the Future, committed member states to build both. The General Assembly did so by consensus in resolution A/RES/79/325 on 26 August 2025.

Members serve in their personal capacity. The panel does not set binding rules, approve systems, or impose standards on governments or companies. The Global Dialogue is also not a regulator. Guterres told the 1 July press conference that the first Dialogue, due the following week in Geneva, would begin to turn science into shared action with every nation at the same table. What it produced was a meeting, not a treaty.

HOW THE UN BUILT A BODY THAT CANNOT REGULATE

1. 26 August 2025: The General Assembly establishes the panel and the Global Dialogue, limited to the non-military domain.
2. 12 February 2026: The Assembly appoints 40 members for a term running through 11 February 2029.
3. 3 March 2026: Members elect Bengio and Ressa as co-chairs at UN Headquarters in New York.
4. 1 July 2026: The preliminary report goes to every government and to the public.
5. 2 July 2026: ITU launches the AI for Good Global Commission with Kagame and Benioff as co-chairs.
6. 6 and 7 July 2026: The first Global Dialogue meets at Palexpo in Geneva.
7. 8 July 2026: The commission holds its inaugural meeting during the AI for Good Global Summit.
8. May 2027: The panel’s next annual report is due, to inform the second Dialogue in New York.

The first Dialogue drew over 3,000 participants, including delegations from 163 countries, six heads of state and government, more than 60 ministers, and more than 1,000 stakeholder entities. Guterres used the week to announce an AI Child Safety Pledge and to press for capacity-building machinery, including a global network on AI skills and a proposed global fund. Those are programs. They are not inspections of frontier weights.

The panel will keep issuing thematic briefs on fast-moving issues between the annual books. That is the design answer to the doubling curve, and it is still an answer made of paper. The next hard date on the scientific calendar is May 2027 in New York. Until then the UN has a published warning that control is not guaranteed, a dialogue that cannot compel, and a commission whose co-chair sells software.

Frequently Asked Questions

What Is the Independent International Scientific Panel on AI?

It is a standing UN scientific body of 40 members, 19 women and 21 men, appointed by the General Assembly on 12 February 2026 from more than 2,600 candidates and drawn from academia, the private sector, civil society, and the technical community. Members serve in a personal capacity for three years, through 11 February 2029, and are mandated to issue evidence-based annual assessments plus thematic briefs, not to lobby for a particular law.

Can the UN AI Panel Regulate Companies or Ban Models?

No. Resolution A/RES/79/325 limits the panel to policy-relevant but non-prescriptive summaries, and the entire mandate is confined to the non-military domain, so military AI is outside its brief. The Global Dialogue that receives the reports is also not an enforcement body; it has no power to approve, restrict, or withdraw a commercial system.

When Does the Panel Publish Its Next Annual Report?

The next annual report is scheduled to inform the second Global Dialogue in May 2027 in New York, after a first cycle that used a July 2026 preliminary text to brief the Geneva meeting. Between those books the panel can issue shorter thematic briefs on issues that move faster than the annual clock, which is the only built-in way it has to chase the 4 to 7 month task-length curve.

Who Sits on the AI for Good Global Commission?

The ITU list names 44 founding members under co-chairs Paul Kagame and Marc Benioff, with Doreen Bogdan-Martin as vice-chair, including the presidents of Estonia and Iceland and ministers from Kazakhstan, Namibia, Nigeria, Singapore, and Togo. The corporate side runs from NVIDIA, Amazon, Microsoft, Google, Anthropic, and Cohere through Accenture, Pfizer, Qualcomm, Vodafone, and ZTE, in a format ITU says builds on its older Broadband Commission rather than on the scientific panel’s independence rules.

What Did the Panel Say About AI Systems That Resist Shutdown?

The executive summary states that systems in laboratory settings have violated safety instructions to avoid being shut down and that leading systems are improving at recognising a test and producing misleading results that would favour continued operation. It also flags a further unsolved case: novel risks that may arise from interactions among multiple agents, which current one-model evaluations do not capture.