?asOf= parameter to see the current catalog state.Jailbreak Resistance
jailbreak-resistance · Frontier safety
The robustness of an AI model's safety training against adversarial prompts crafted to elicit policy-prohibited outputs — distinct from alignment (which concerns the model's goals) and from baseline safety training (which concerns the model's defaults).
Definition & scope
Jailbreak resistance is the operational counterpart to alignment. A model can be 'aligned' in the sense of internalising its principal's intent at training time and still be 'jailbreakable' in the sense that adversarial prompting recovers prohibited behaviours. The attack literature is extensive: roleplay-framing attacks (DAN-style prompts, 2022-2023), encoding attacks (Wei et al. 2023, 'Jailbroken: How Does LLM Safety Training Fail?'), gradient-based suffix attacks (Zou et al. 2023, 'Universal and Transferable Adversarial Attacks on Aligned Language Models'), many-shot jailbreaking (Anil et al. 2024, Anthropic, exploiting long context), and persuasion-style attacks (Zeng et al. 2024, 'How Johnny Can Persuade LLMs to Jailbreak Them'). Industry defences (constitutional classifiers, RLHF + constitutional AI, output filters, multi-stage safety pipelines) are improving but no model has demonstrated full robustness; the white-hat assumption is that adequately-resourced attackers can find a working jailbreak for any current frontier model. Governance relevance: EU AI Act Art. 55(1)(a) adversarial-testing requirement directly targets jailbreak resistance; the testing methodology must include adversarial probing. UK AISI evaluations include public-domain + novel jailbreak probes. NIST AI RMF GenAI Profile §2.6 'Information Security' addresses adversarial robustness. Industry-side frameworks (Anthropic RSP, OpenAI Preparedness, DeepMind FSF) treat jailbreak resistance as one input to capability-tier safeguards — at high CBRN-uplift capability, jailbreak resistance becomes load-bearing for deployment safety.
Use in governance
How instruments operationalise this concept
| Instrument | Jurisdiction | Status |
|---|---|---|
| EU AI Act | EU | in force |
| NIST AI RMF Generative AI Profile | US | in force |
| G7 Hiroshima AI Process Code of Conduct | G7 | in force |
Appears in topic articles
Editorial note
Distinguish jailbreak resistance (robustness to adversarial elicitation of prohibited outputs) from alignment (whether the model's goals match the principal's) and from prompt injection (whether untrusted content can hijack the instruction channel). All three are necessary but none is sufficient for deployment safety.
See also
Further reading
Sources on the broader topics this concept relates to — complementing, not standing in for, the primary sources cited inline above. 66 academic & grey-literature sources; catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways Peer-reviewed✦ AIReviews biosecurity and dual-use risks at the AI-synthetic-biology interface and maps governance pathways for emerging catastrophic threats.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Two types of AI existential risk: decisive and accumulative Peer-reviewed✦ AIDistinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios.
- Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence Peer-reviewed✦ AIArgues international law imposes a precautionary-principle obligation on states to regulate AI to mitigate the threat of human extinction.
- Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility Peer-reviewed✦ AIAnalyzes how AI-driven detection/concealment in nuclear arsenals reshapes strategic stability and proliferation risk, with governance implications.
- International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty Preprint✦ AIProposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable.
- Managing extreme AI risks amid rapid progress Peer-reviewed✦ AIWarns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Evaluating Frontier Models for Dangerous Capabilities Preprint✦ AIPilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating.
+ 54 more across this concept's topics — see the literature index.
References
The primary instrument sources behind the article's classifications.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/jailbreak-resistance — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here