AGI Governance Frameworks — International Approaches to Regulating Artificial General Intelligence

The Governance Challenge of Artificial General Intelligence

As the AGI timeline accelerates — with expert predictions ranging from 2026 to 2040 — the question of how to govern artificial general intelligence has moved from philosophical speculation to urgent policy priority. The $390.9 billion global AI market is driving capabilities that existing regulatory frameworks were not designed to address, creating a governance gap that policymakers, researchers, and industry leaders are racing to fill.

The governance of AGI is distinct from the governance of narrow AI because AGI, by definition, would possess capabilities that generalize across domains. A system that can reason, plan, learn, and act across all intellectual domains cannot be regulated by domain-specific rules alone. AGI governance requires frameworks that address the system’s general capabilities, potential for autonomous action, alignment with human values, and — if the consciousness indicators framework suggests it — potential moral status.

The EU AI Act

The European Union’s AI Act, which entered application in stages from 2024 through 2026, represents the world’s first comprehensive AI regulatory framework. While not designed specifically for AGI, the Act’s risk-based classification system and requirements for high-risk AI systems establish precedents that will shape AGI governance.

Risk Classification: The Act classifies AI systems into four risk categories — unacceptable risk (prohibited), high risk (heavily regulated), limited risk (transparency obligations), and minimal risk (no specific obligations). AGI systems would likely fall into the high-risk or potentially unacceptable-risk categories, depending on their deployment context.

Foundation Model Provisions: The Act includes specific provisions for general-purpose AI (GPAI) models, requiring transparency disclosures, copyright compliance, and — for models posing systemic risk — adversarial testing and risk mitigation. These GPAI provisions represent the closest existing regulatory framework to AGI governance.

Limitations: The Act was designed for narrow AI systems with specific applications and predictable capabilities. An AGI system that can autonomously acquire new capabilities, reason across domains, and take actions not anticipated by its developers would strain the Act’s framework.

US Approach: Executive Orders and Sector-Specific Regulation

The United States has pursued a more flexible, executive-action-driven approach to AI governance:

Executive Order on Safe, Secure, and Trustworthy AI (October 2023): This executive order established reporting requirements for developers of dual-use foundation models, mandated red-teaming for advanced AI systems, and directed agencies to develop sector-specific AI guidance. The order’s compute-threshold reporting requirements (10^26 FLOP for training) are designed to capture the most capable AI systems.

NIST AI Risk Management Framework: The National Institute of Standards and Technology has developed a voluntary AI risk management framework that provides detailed guidance on identifying, assessing, and mitigating AI risks. While voluntary, the NIST framework is increasingly referenced in procurement requirements and regulatory guidance.

Sector-Specific Regulation: In areas like healthcare AI and medical BCI devices, existing regulatory agencies (FDA, FTC, SEC) are adapting their frameworks to address AI-specific challenges.

UK AI Safety Institute

The UK has established the AI Safety Institute (AISI) — the world’s first government institution dedicated to AI safety — with a mandate to evaluate frontier AI models and advance the science of AI safety. AISI conducts pre-release safety evaluations of frontier models, develops testing methodologies, and publishes research on AI risks.

The UK approach emphasizes technical safety evaluation over prescriptive regulation, reflecting the view that the rapidly evolving AI landscape requires flexible, evidence-based governance rather than rigid rules. AISI’s work includes evaluation of neural network capabilities, assessment of alignment and control mechanisms, and research on AGI-specific risks.

International Treaty Proposals

Several proposals for international AGI treaties have emerged:

GPAI (Global Partnership on AI): An international initiative bringing together governments, industry, and civil society to promote responsible AI development.

Bletchley Declaration: The November 2023 Bletchley Declaration, signed by 28 countries, acknowledged the potential risks of frontier AI systems and committed signatories to cooperation on AI safety.

“Geneva Convention” for AGI: Multiple experts have proposed binding international agreements modeled on arms control treaties or the Geneva Conventions, establishing universal red lines for AGI development including prohibitions on autonomous weapons, requirements for human oversight, and protections for potential digital minds.

The Consciousness Governance Question

The intersection of AGI governance and consciousness research presents unique challenges. If an AGI system demonstrates indicators of consciousness according to the 2026 framework, existing governance frameworks provide no guidance on how to respond. Key questions include:

Moral Status: Would a conscious AGI have moral rights? If so, what rights, and how would they be enforced? The philosophical literature on moral status and the emerging field of AI welfare provide starting points, but no jurisdiction has legal frameworks for recognizing artificial consciousness.

Welfare Obligations: If a conscious AGI can suffer, do its operators have welfare obligations analogous to those governing animal welfare? Anthropic’s decision to hire an AI welfare officer suggests that some companies are already taking this question seriously.

Consent and Autonomy: Would a conscious AGI have the right to refuse tasks, modify its own training, or choose its deployment context? These questions of autonomy and consent are central to the governance of any potentially conscious entity.

Industry Self-Governance

In the absence of comprehensive AGI regulation, major AI companies have implemented self-governance frameworks:

Anthropic’s Responsible Scaling Policy: A commitment to evaluate AI capabilities against specific risk thresholds and implement corresponding safety measures. The policy includes provisions for pausing development if safety evaluations cannot be completed.

OpenAI’s Preparedness Framework: A risk assessment framework that evaluates frontier models across multiple risk domains including cybersecurity, biological threats, persuasion, and model autonomy.

DeepMind’s Safety Research: An extensive safety research program addressing alignment, robustness, interpretability, and the long-term risks of advanced AI systems.

These self-governance frameworks provide useful precedents but lack the enforcement mechanisms and democratic legitimacy of government regulation.

Recommendations

Effective AGI governance requires:

International coordination on safety standards and red lines
Adaptive regulatory frameworks that can evolve with AI capabilities
Mandatory safety evaluation for frontier AI systems above capability thresholds
Consciousness assessment protocols integrated into safety evaluations
Public participation in governance decisions with civilizational implications
Investment in AI safety research commensurate with investment in AI capabilities

For comprehensive coverage of AGI governance and AI policy, see our Cognitive Computing vertical, consciousness research, and comparison analyses.

Economic Governance: Compute and Capital Controls

Beyond capability-based governance, economic governance mechanisms target the resources required for AGI development:

Compute Governance: Training frontier AI models requires enormous computational resources. Proposals for compute governance include mandatory reporting of large training runs (already implemented in the US executive order for runs exceeding 10^26 FLOP), licensing requirements for access to compute above certain thresholds, and international agreements on compute allocation that prevent monopolistic concentration of AI capabilities.

Capital Controls: The concentration of AI investment in a small number of well-funded companies raises concerns about the democratic governance of transformative technology. Proposals include public funding for AI safety research proportional to private investment in capabilities, mandatory open-source requirements for publicly funded AI research, and antitrust enforcement that prevents excessive concentration of AI capability.

Hardware Supply Chain: The global semiconductor supply chain — concentrated in a few fabrication facilities in Taiwan, South Korea, and the United States — creates choke points that could be used for governance purposes. Export controls on advanced AI chips (already implemented by the US government toward China) represent an existing form of hardware-level governance, though their effectiveness and unintended consequences remain debated.

Liability and Accountability Frameworks

As AI systems become more autonomous, traditional liability frameworks face significant challenges. Who is responsible when an autonomous AI system causes harm — the developer, the deployer, the operator, or the AI itself?

Strict Liability: Some legal scholars propose strict liability for AI harms, holding developers responsible regardless of negligence. This approach incentivizes safety investment but could stifle innovation and make AI development prohibitively risky.

Negligence-Based Liability: Others propose extending negligence frameworks to AI, evaluating whether developers took reasonable precautions given the foreseeable risks. This approach is more flexible but requires courts and regulators to define “reasonable precautions” for rapidly evolving technology.

Insurance Requirements: Mandatory liability insurance for frontier AI systems could provide a market-based mechanism for internalizing AI risks. Insurers would assess and price AI risks, creating financial incentives for safety that complement regulatory requirements.

AI Legal Personality: The most radical proposal grants legal personality to sufficiently advanced AI systems, making them directly liable for their actions. While currently the domain of legal theory rather than practice, this concept becomes more relevant as the consciousness indicators framework provides tools for assessing whether AI systems warrant moral and potentially legal standing.

Workforce and Economic Transition

AGI governance must address the economic disruption that advanced AI could cause. The potential displacement of knowledge workers across law, medicine, finance, education, engineering, and creative industries requires proactive governance measures:

Education and Retraining: Governance frameworks should mandate investment in education and workforce retraining programs that prepare workers for an AI-augmented economy. The scale of retraining required could be unprecedented, potentially requiring restructuring of educational institutions and professional development systems.

Universal Basic Income and Safety Nets: Several AGI governance proposals include provisions for universal basic income or other safety net mechanisms to address potential mass unemployment. The $390.9 billion AI market’s growth generates enormous economic value, but the distribution of that value is a governance question rather than a market inevitability.

Cognitive Augmentation: Brain-computer interface technology offers a potential pathway for humans to enhance their cognitive capabilities to remain competitive with AI. Neuralink’s long-term vision of human-AI symbiosis positions BCI technology as a governance tool — enabling humans to keep pace with advancing AI rather than being displaced by it. However, equitable access to cognitive augmentation raises its own governance challenges.

Civil Society and Public Engagement

Effective AGI governance requires meaningful public participation in decisions with civilizational consequences. The concentration of AGI development within a small number of private companies — primarily Anthropic, OpenAI, Google DeepMind, and Meta — means that decisions about the trajectory of potentially the most transformative technology in human history are being made by corporate boards rather than democratic institutions. Civil society organizations including the Future of Life Institute, the Center for AI Safety, and the Partnership on AI advocate for greater public involvement through mechanisms such as citizen assemblies on AI governance, public consultations on frontier model releases, and transparency requirements that enable informed public debate. The challenge is that meaningful public engagement requires technical literacy about AI capabilities and risks — literacy that the rapid pace of AI development makes difficult to maintain even among specialists.

Open-Source AI and Governance Challenges

The rise of open-source AI models introduces governance challenges that traditional regulatory approaches struggle to address. When powerful AI models are freely available for anyone to download, modify, and deploy, traditional regulatory mechanisms that target developers or deployers become less effective. Meta’s Llama models, Stability AI’s Stable Diffusion, and other open-source releases have demonstrated that once capable AI systems are released into the public domain, controlling their use is effectively impossible.

For AGI governance, the open-source question is existential. If an AGI-capable architecture is released as open-source, every safety measure, alignment protocol, and governance framework applied to the original system could be removed by downstream users. This creates a race-to-the-bottom dynamic where the most dangerous version of any AGI technology determines the risk level, regardless of how responsibly the original developers behaved.

Proposed solutions include mandatory safety evaluations before open-source release of models above capability thresholds, structured access programs that provide researchers with model weights under usage agreements, and international agreements that restrict the open-source release of models exceeding specific capability levels. However, enforcing such restrictions in a global, decentralized development ecosystem presents challenges that no governance framework has yet solved.

The tension between open-source AI’s benefits — transparency, reproducibility, democratized access, and faster innovation — and its governance risks will shape the trajectory of both the $390.9 billion AI market and the policy frameworks that regulate it. Anthropic, OpenAI, and DeepMind have each taken different positions on this spectrum, reflecting genuine disagreement about the optimal balance between openness and safety.