The field of Artificial Intelligence (AI) has long been divided into Artificial Narrow Intelligence (ANI) and Artificial General Intelligence (AGI). However, most AI governance strategies tend to concentrate solely on ANI, often neglecting AGI, which many view as the pinnacle of the field. Recently, a significant report has been released that acknowledges the importance of AGI and outlines proposals for its governance at the United Nations level. In this post, I intend to delve into this new report and share my perspective on its findings.
The report is titled “Governance of the Transition to Artificial General Intelligence: Urgent Considerations for the UN General Assembly”, prepared by the High-Level Expert Panel on AGI, which was convened by the UN Council of Presidents of the General Assembly (UNCPGA) during their 2024 meeting in Seoul. At that meeting, the Council agreed to commission a report addressing the risks, threats, and opportunities posed by AGI. That report has now been formally submitted to the President of the General Assembly, with discussions on its implementation already underway. Although official briefings are expected in the coming months, the report’s early release invites timely engagement from the broader community.
During my time at the Millennium Project, I had the chance to see this report take shape through different drafts and discussions. It was clear even then that this wasn’t going to be just another checklist of AI principles. It’s a focused, grounded document that treats AGI as something qualitatively different and potentially imminent.
The High-Level Expert Panel behind the report consists of the following members. Jerome Glenn (USA), Chair, The CEO of Millenium Project and includes leading international experts, such as Renan Araujo (Brazil), Yoshua Bengio (Canada), Joon Ho Kwak (Republic of Korea), Lan Xue (China), Stuart Russell (UK and USA), Jaan Tallinn (Estonia), Mariana Todorova (Bulgaria) and José Jaime Villalobos (Costa Rica). As we can see, the Panel consist of many experts that we know from their work on AI.
As the title of the report suggests, the panel outlines a series of consequences that may follow from global inaction. These are presented not as distant hypotheticals but as plausible outcomes if early governance measures are neglected.
Here below is the list provided by the Panel:
1. Irreversible Consequences.
Once AGI is achieved, its impact may be irreversible. With many frontier forms of AI already showing deceptive and self-preservation behaviour, and the push towards more autonomous, interacting, self-improving AIs integrated with infrastructures, the impacts and trajectory of AGI can plausibly end up being uncontrollable. If that happens, there may be no way to return to a state of reliable human oversight.
2. Weapons of Mass Destruction.
AGI could enable some states and malicious nonstate actors to build chemical, biological, radiological, and nuclear weapons. Moreover, large, AGI-controlled swarms of lethal autonomous weapons could themselves constitute a new category of WMDs.
3. Critical Infrastructure Vulnerabilities.
Critical national systems (e.g., energy grids, financial systems, transportation networks, communication infrastructure, and healthcare systems) could be subject to powerful cyberattacks launched by or with the aid of AGI.
4. Power Concentration, Global Inequality, and Instability.
Uncontrolled AGI development and usage could exacerbate wealth and power disparities on an unprecedented scale. If AGI remains in the hands of a few nations, corporations, or elite groups, it could entrench economic dominance and create global monopolies over intelligence, innovation, and industrial production.
5. Existential Risks.
AGI could be misused to create mass harm or developed in ways that are misaligned with human values; it could even act autonomously beyond human oversight, evolving its own objectives according to self-preservation goals already observed in current frontier AIs. AGI might also seek power as a means to ensure it can execute whatever objectives it determines, regardless of human intervention.
6. Loss of Extraordinary Future Benefits for All of Humanity.
Properly managed AGI promises improvements in all fields, for all peoples, from personalized medicine, curing cancer, and cell regeneration, to individualized learning systems, ending poverty, addressing climate change, and accelerating scientific discoveries with unimaginable benefits. Ensuring such a magnificent future for all requires global governance, which begins with improved global awareness of both the risks and benefits. The United Nations is critical to this mission.
Following these warnings, the report clearly articulates the purpose of UN-led governance, which is to ensure that the development and use of AGI align with global human values, security, and developmental goals.
To support this, it outlines five key priorities that should guide international action.
1. Advance research on AI alignment and control to identify technical methods for guiding increasingly capable systems.
2. Establish frameworks to ensure AGI is developed responsibly, with strong security safeguards, transparency, and alignment with human values.
3. Create governance structures for AGI deployment to prevent misuse, guarantee equitable access, and maximise benefits while reducing risks.
4. Support long-term visions for beneficial AGI, including new models for social, environmental, and economic development.
5. Build an inclusive platform for international cooperation, set global standards, and encourage compliance through legal mechanisms that foster trust and ensure shared access to AGI’s potential.
To respond to the uncertainties surrounding AGI development, the report calls for coordinated international action led by the UN. It proposes the creation of a Global AGI Observatory to monitor progress in AGI research and issue early warnings about potential security risks. Alongside this, it recommends establishing an international certification system to promote secure, transparent, and ethically aligned development practices. A UN Framework Convention on AGI is also proposed, aiming to set shared objectives, define risk thresholds, and create adaptable governance tools such as licensing regimes and joint research initiatives. Finally, the report suggests launching a feasibility study for a dedicated UN agency on AGI, acknowledging that the complexity of AGI governance may require an institutional structure beyond existing models.
These proposals arrive at a moment when AI systems are advancing rapidly, regardless of whether they are explicitly framed as steps toward AGI or not. Let us consider the recent advancements in AI, particularly in large language models like GPT-4 and their integration into autonomous, evolutionary agents such as DeepMind’s AlphaEvolve. These developments increasingly reflect capacities often associated with general intelligence (encapsulating generality, specialised intelligence, self-improvement and learning from experiences). They may not yet be AGI, but the threshold is getting closer with each breakthrough. In fact, some AI companies make no effort to conceal their intentions, many explicitly frame the creation of AGI as central to their mission.
In my recent article, I categorised AGI as a wicked problem, following the framework established by Rittel and Webber’s Wicked Problems Theory. One key insight from this framework is the necessity of multidisciplinary collaboration, a point the report rightly underscores. Another key insight is that, since AGI lacks a definitive formulation, there is no consensus on its definition or the exact path to achieving general intelligence in machines. However, among experts and leading research groups pursuing AGI, the underlying ethos (or the core aim) is clear: AGI is expected to reach or even surpass human intelligence and can be deployed in various domains, with general-purpose capabilities. Additionally, there is no natural endpoint to its evolution. It could be the result of human ingenuity, either by a single individual’s effort or a collective endeavour, but it could also be a breakthrough made by AI itself.
AGI is not just flexible or multi-purpose (like GPAI models/systems under the EU’s AI Act); it is expected to operate autonomously, without needing human input to initiate or carry out actions. Recognising this capacity means taking its potential realisation seriously and acting accordingly. Since the idea of AI reaching human-level first emerged, experts have offered endless predictions about when it might arrive. None of these predictions can be verified until AGI actually emerges, so until that point is reached, they remain speculative. Some believe recent advances signal a real breakthrough. Others are more sceptical, suggesting these developments may have little bearing on AGI’s eventual emergence. Whether the transition happens incrementally or suddenly remains uncertain, but what’s clear is that each new advancement makes the prospect feel less like distant speculation. In either case, the stakes are too high to ignore the prospect of AGI.
Given these advancements and the growing plausibility of AGI, it is essential to take ethical, forward-looking governance measures while there is still time. As the Millennium Project, other AGI governance articles, and I stated elsewhere, the Collingridge Dilemma reminds us that the transition from ANI to AGI may be the only viable window for intervention. Once AGI is realised, effective governance may no longer be feasible due to a lack of time, resources, or control. This is precisely why a high-level expert panel is needed to identify the issues the UN should begin considering now.
We’ll see what the future holds. Perhaps the UN General Assembly will rise to the occasion and turn this report into meaningful global coordination. Or, as David Wood has warned, cooperation may falter, and the political will required for serious AGI governance may never materialise. Either way, something has shifted, and now it’s harder to ignore the shift. In fact, for the first time, the OECD introduces AGI into its AI governance vocabulary. AGI research is no longer fringe speculation, and AI governance is no longer confined to technical compliance. Scholars in law, philosophy, psychology, and beyond are starting to grapple seriously with what it means to share intelligence with systems that may soon think, reason, and act beyond human prompting. Whether we frame AGI as a wicked problem, a moral experiment, or a governance conundrum, one thing is clear. There’s no going back to a time when we could treat this as someone else’s concern.
Yeliz Figen Döker is a PhD researcher at the European University Institute, where she works on AGI regulation. She is the co-founder and Head of Science Fiction at DigiCon.
