Pinned snapshot. This article is rendered from the catalog state captured at (closest snapshot at-or-before your requested date of ). The live version may differ — drop the
?asOf= parameter to see the current catalog state.Governance of model capabilities that could cause mass casualties or civilisational-scale harms (CBRN uplift, autonomous replication, deceptive alignment). Distinct from EU AIA 'systemic risk' which targets market-scale rather than catastrophic-scale harms.
Definition & scope
The cross-jurisdiction picture below shows how each of 45 tracked instruments treats this topic. The patterns vary substantially — and 23 regimes are silent, leaving gaps that future policy work could address.
Coverage across jurisdictions
Historical primacy & cross-jurisdiction tension
First addressed by DoD Responsible AI Strategy and Implementation Pathway on (implicit). Subsequent regimes have either codified, diverged from, or remained silent on this baseline.
- Forum-shoppingExecutive Order 14110 on Safe, Secure, Trustworthy AI↔Executive Order 14179 — Removing Barriers to American Leadership in AI
- Forum-shoppingG7 Hiroshima AI Process Code of Conduct↔Interim Measures for Generative AI Service Management
- Forum-shoppingBletchley Declaration on AI Safety↔OECD AI Principles (Recommendation)
Compare jurisdictions: EU vs US · EU vs UK · EU vs CN
Enforcement & impact
Silent regimes — gap signal
Instruments that do not address Catastrophic & Existential Risk — candidates for future policy work.
- Executive Order 14179 — Removing Barriers to American Leadership in AIUS
- Interim Measures for Generative AI Service ManagementCN
- OECD AI Principles (Recommendation)OECD
- Council of Europe Framework Convention on AIcouncil_of_europe
- India Digital Personal Data Protection Act + AI Advisory (MEITY)IN
- ASEAN Guide on AI Governance and EthicsASEAN
- African Union Continental AI StrategyAfrican_Union
- Singapore Model AI Governance Framework for Generative AISG
- Japan METI AI Guidelines for BusinessJP
- General Data Protection Regulation (GDPR)EU
- OMB Memorandum M-24-10 (Advancing Governance, Innovation, and Risk Management for Agency Use of AI)US
- GSA Generative AI and Specialized Computing Infrastructure Acquisition Resource GuideUS
- FedRAMP AI Cloud Procurement GuidanceUS
- DFARS Subpart 252.204 (Safeguarding Covered Defense Information and Cyber Incident Reporting)US
- California SB 243: Companion ChatbotsUS
- California SB 942: AI Transparency ActUS
- Revised Product Liability Directive (Directive (EU) 2024/2853)EU
- UNESCO Recommendation on the Ethics of Artificial IntelligenceUNESCO
- Directive (EU) 2024/2831 on improving working conditions in platform workEU
- Provisions on the Administration of Deep Synthesis of Internet Information ServicesCN
- TAKE IT DOWN Act (Tools to Address Known Exploitation by Immobilizing Technological Deepfakes on Websites and Networks Act)US
- Italy Law No. 132/2025 on Artificial Intelligence (Legge 23 settembre 2025, n. 132)IT
- Japan AI Promotion Act (Act on the Promotion of Research, Development and Utilization of AI-Related Technologies)JP
See also
Further reading
18 academic & grey-literature sources bearing on this topic — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways Peer-reviewed✦ AIReviews biosecurity and dual-use risks at the AI-synthetic-biology interface and maps governance pathways for emerging catastrophic threats.
- Two types of AI existential risk: decisive and accumulative Peer-reviewed✦ AIDistinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios.
- Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence Peer-reviewed✦ AIArgues international law imposes a precautionary-principle obligation on states to regulate AI to mitigate the threat of human extinction.
- Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility Peer-reviewed✦ AIAnalyzes how AI-driven detection/concealment in nuclear arsenals reshapes strategic stability and proliferation risk, with governance implications.
- International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty Preprint✦ AIProposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable.
- Managing extreme AI risks amid rapid progress Peer-reviewed✦ AIWarns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D.
- Verification methods for international AI agreements Preprint✦ AISurveys '10 verification methods that could detect... unauthorized AI training... and unauthorized data centers', mapping the technical basis for compute-disclosure regimes.
- AI and biosecurity: The need for governance Peer-reviewed✦ AIArgues 'governments should evaluate advanced [biological] models and if needed impose safety measures' to mitigate AI-enabled biosecurity catastrophic risk.
- From Principles to Rules: A Regulatory Approach for Frontier AI Preprint✦ AIRecommends frontier-AI regulation begin with high-level safety principles and migrate to detailed rules (e.g., mandated dangerous-capability evaluations) as regulatory capacity matures.
- Model evaluation for extreme risks Preprint✦ AIProposes "dangerous capability evaluations" and alignment evaluations of frontier models so developers and policymakers can make "responsible decisions about model training, deployment, and security".
- Frontier AI Regulation: Managing Emerging Risks to Public Safety Preprint✦ AIArgues "industry self-regulation is an important first step" but "government intervention will be needed", proposing safety standards, registration and reporting, and compliance mechanisms.
- Can large language models democratize access to dual-use biotechnology? Preprint✦ AIRed-team exercise finding LLM chatbots "may also confer easy access to dual-use technologies capable of inflicting great harm" and could make pandemic-class agents more widely accessible.
- Agentic AI System PreprintYao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y. (2022), 'ReAct: Synergizing Reasoning and Acting in Language Models.'
- Tool-Use Safety PreprintWallace, E., et al. (2024), 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions' (OpenAI) — the canonical industry articulation of instruction-channel hierarchy as a tool-use-safety defence.
- Jailbreak Resistance PreprintZou, A., Wang, Z., Kolter, J. Z., Fredrikson, M. (2023), 'Universal and Transferable Adversarial Attacks on Aligned Language Models' — the canonical demonstration that gradient-based suffix attacks transfer across aligned LLMs.
- Chain-of-Thought Monitoring PreprintKorbak, T., Balesni, M., Barnes, E., Bengio, Y., et al. (2025), 'Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.' arXiv:2507.11473.
- Artificial Intelligence Research institute✦ AIUS National Academies' AI consensus-study hub.
- Beyond P(doom) for AI Risk: Quantifying Uncertainty Without Probability Research institute✦ AIArgues AI-risk assessment should characterise structured uncertainty instead of collapsing to a single 'probability of doom' number.
References
The primary instrument sources behind the article's classifications.
- EU-AIA-2024: Art. 51 + Recital 32 — systemic risk overlaps with but does not fully cover catastrophic-risk framing
- US-EO-14110: §4.2(a)(ii) — CBRN + autonomous replication explicitly named
- UK-WHITEPAPER-2023: AISI remit covers frontier-model evaluation; not in white paper text
- G7-HIROSHIMA: Code §1 + §3 — explicit risk-identification including CBRN
- UN-RES-2024: Notes 'shared concerns' but no operative catastrophic-risk text
- NIST-AI-RMF: Map 1.1 risk classification covers catastrophic via 'societal' impact tier; GenAI Profile (2024) adds explicit content
- BLETCHLEY-2023: Declaration §3-5 (substantial risks from frontier AI, including catastrophic harm)
- SEOUL-2024: Frontier AI Safety Commitments §1: identify thresholds for severe risks pre-deployment
- NIST-AI-RMF-GENAI: NIST AI 600-1 §3.1 CBRN Information Uplift; §3.3 Dangerous, Violent, or Hateful Content
- CA-SB-1047: Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage
- BR-AIBILL-2024: PL 2338/2023 Art. 14 (excessive-risk AI applications — explicit prohibition + risk-tier framework)
- ANTHROPIC-RSP-2024: RSP v2 §3 — ASL-3 / ASL-4 capability thresholds explicitly target CBRN uplift + autonomous-replication
- OPENAI-PREPAREDNESS-2023: Preparedness Framework risk-tier matrix — Critical tier explicitly targets CBRN, cyber, persuasion, autonomy
- DEEPMIND-FSF-2024: FSF Critical Capability Levels (CCL) — explicit thresholds for autonomy, biosecurity, cyber, persuasion
- META-FRONTIER-2024: Framework critical-risk tier — commit to halt training pre-mitigation if reached
- UK-US-AISI-MOU-2024: Joint evaluation scope encompasses CBRN + autonomy uplift questions; MoU text does not enumerate explicit thresholds
- WH-VOLUNTARY-2023: Commitments §1 references CBRN + bio risks via 'most significant societal risks'; not threshold-explicit
- EU-GPAI-COP-2025: Chapter 3 systemic-risk-tier capability evaluations + serious-incident reporting + model-weight access controls (Art. 55 substrate)
- DOD-RAI-2022: Ethical Principle 'Reliable' + Tenet 4 (Requirements Validation) — JCIDS gating addresses mission-risk; DoDD 3000.09 separately governs autonomy-in-weapons LAWS-specific catastrophic-risk decisions
- CA-SB-53: Bus. & Prof. Code § 22757.11 (definition) operationalized by §§ 22757.12 (framework) + 22757.13 (critical-safety-incident reporting to CalOES)
- NY-RAISE-2025: N.Y. Gen. Bus. Law § 1421(1) requires a large developer to implement and conspicuously publish a written safety and security protocol governing the risk of 'critical harm' from its frontier models, and § 1421(4) requires disclosure of safety incidents within 72 hours; § 1420(7) defines critical harm (100+ deaths/serious injuries or $1B damage via CBRN weapons or autonomous model conduct). NOTE: the floor-text § 1421(2) deployment PROHIBITION was struck by the chapter amendment enacted Mar. 27, 2026 (S8828/A9449), which reoriented the Act to a transparency-and-reporting regime; this cell tracks the RETAINED safety-protocol + incident-reporting duties, not a deployment ban.
- UN-GDC-2024: GDC Objective 5, paras 55(a) and 56(a) (A/RES/79/1, Annex I)
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Verify the year + paste-and-refine. Primary source linked in BibTeX/RIS note.
Persistent identifier: https://policywindow.org/wiki/catastrophic-risk — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
Source: Edit on GitHub (search for `catastrophic_risk`)
22 instruments tracked.