?asOf= parameter to see the current catalog state.Red-Team Evaluation
red-team-evaluation · Frontier safety
Structured adversarial probing of an AI model's capabilities and behaviour before deployment, designed to elicit failures that ordinary evaluation would miss.
Definition & scope
Red-team evaluation originated in cybersecurity (penetration testing) and was adapted to AI by the 2022 DEF CON Generative Red Team event and later codified in the 2023 White House voluntary commitments. EU AI Act Art. 55(1)(a) requires adversarial testing for general-purpose AI models with systemic risk. US EO 14110 §4.2(a)(i) required reporting of red-team results for foundation models above the compute threshold (rescinded under EO 14179). G7 Hiroshima Code §1 calls for 'adversarial testing prior to and throughout deployment.' Anthropic, OpenAI, and Google DeepMind each maintain internal red-team programs with public methodology disclosures. Governance disputes centre on: (1) WHO must red-team (provider, independent third-party, government); (2) WHAT capabilities are in scope (CBRN uplift, autonomous replication, election manipulation, etc.); (3) WHO sees the results (provider only, regulator under confidentiality, public); (4) WHAT triggers re-evaluation after deployment.
Locus of dispute: WHO must red-team (provider, independent third-party, regulator), WHAT capabilities are in scope (CBRN uplift, autonomous replication, election manipulation), and WHO sees the results (provider only, regulator under confidentiality, public)? Field convergence post-Seoul 2024 is slow.
Use in governance
How instruments operationalise this concept
| Instrument | Jurisdiction | Status |
|---|---|---|
| EU AI Act | EU | in force |
| Executive Order 14110 on Safe, Secure, Trustworthy AI | US | partial |
| G7 Hiroshima AI Process Code of Conduct | G7 | in force |
| UK Pro-Innovation Approach to AI Regulation (White Paper) | UK | in force |
| Anthropic Responsible Scaling Policy (RSP) v2 | US | in force |
| OpenAI Preparedness Framework | US | in force |
| Google DeepMind Frontier Safety Framework | US | in force |
| Meta Frontier AI Framework | US | in force |
| UK-US AI Safety Institute Memorandum of Understanding | global | in force |
| White House Voluntary AI Commitments | US | in force |
| Singapore Model AI Governance Framework for Generative AI | SG | in force |
| Japan METI AI Guidelines for Business | JP | in force |
Appears in topic articles
Editorial note
Distinguish from 'evaluation' (general benchmark-style measurement) and 'audit' (post-hoc third-party review). Red-teaming is specifically pre-deployment + adversarial-intent.
See also
Further reading
Sources on the broader topics this concept relates to — complementing, not standing in for, the primary sources cited inline above. 70 academic & grey-literature sources; catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Defending Compute Thresholds Against Legal Loopholes Preprint✦ AIIdentifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds.
- The Current Landscape of Deepfake Legislation in the United States Peer-reviewed✦ AIThematic analysis of 319 state deepfake bills (2019-2024) finds a fragmented patchwork concentrated on political and sexually-explicit content.
- Reimagining U.S. Tort Law for Deepfake Harms: Comparative Insights from China and Singapore Peer-reviewed✦ AIArgues fragmented US tort doctrines (defamation, publicity, IIED) are ill-suited to deepfake harms and draws remedial lessons from Chinese and Singaporean law.
- A Teleological Interpretation of the Definition of DeepFakes in the EU Artificial Intelligence Act—A Purpose-Based Approach to Potential Problems With the Word 'Existing' Peer-reviewed✦ AIWarns a narrow reading of 'existing' in the AI Act's deepfake definition could exclude synthetic media from transparency duties, urging a teleological interpretation.
- Audio deepfakes and the regulation of the landlords of creativity Peer-reviewed✦ AIArgues US, EU and Chinese regimes fail to assign audio-deepfake liability to 'landlords of creativity' (foundation-model providers) and proposes holding them accountable.
- Human detection of political speech deepfakes across transcripts, audio, and video Peer-reviewed✦ AIExperiments show "audio and visual information enables more accurate discernment than text alone" — humans rely more on how something is said than on transcript content.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Computing Power and the Governance of Artificial Intelligence Preprint✦ AIArgues compute is a uniquely governable lever because it is "detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain".
- Training Compute Thresholds: Features and Functions in AI Regulation Preprint✦ AIFinds "training compute currently is the most suitable metric to identify GPAI models", but thresholds should only trigger further scrutiny, not determine risk measures alone.
+ 58 more across this concept's topics — see the literature index.
References
The primary instrument sources behind the article's classifications.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/red-team-evaluation — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here