{"$schema":"https://policywindow.org/wiki/evidence.json","name":"Policy Window social-science evidence map","description":"Per-topic evidence base across the published catalog: is the harm empirically real (problem-reality) and is there rigorous evidence the governance works (governance-efficacy), each epistemic-status-labelled and sourced. The aggregate surfaces the 'second silence' — the evidence that the rules work is itself largely missing.","docs":"https://policywindow.org/wiki/evidence","method":"https://policywindow.org/wiki/methodology","dimensions":{"problem-reality":"Is the underlying harm/phenomenon empirically real?","governance-efficacy":"Is there rigorous evidence that governing this topic reduces the harm / achieves its aim?"},"epistemicStatus":{"established":"robust + replicated evidence","contested":"genuine empirical disagreement in the literature","thin":"little evidence yet (early / sparse)","absent":"no evidence base exists"},"summary":{"publishedTopics":23,"topicsWithEvidence":23,"coverage":1,"problemReality":{"established":7,"contested":12,"thin":4,"absent":0},"governanceEfficacy":{"established":0,"contested":0,"thin":6,"absent":17},"efficacyThinOrAbsent":23,"efficacyThinOrAbsentShare":1,"realHarmUnprovenGovernance":19,"headline":"Of 23 evidence-reviewed topics, 23 have thin or absent evidence that governance actually works, and 19 pair a real-or-contested harm with empirically unproven governance."},"topics":[{"code":"agentic_systems_governance","label":"Agentic AI Governance","kind":"capability","empiricalConsensus":"emerging","realHarmUnprovenGovernance":false,"problemReality":{"epistemicStatus":"thin","finding":"The capability that agentic governance targets — autonomous multi-step action — is real and rapidly, measurably advancing: METR finds the task length AI agents complete at 50% reliability has doubled roughly every seven months for the past six years (about 50 minutes for frontier 2025 models), and the UK AI Security Institute's first Frontier AI Trends Report (Dec 2025, >30 systems) reports models now finish hour-long software tasks >40% of the time versus <5% in late 2023. The distinct realized HARM from agency (as opposed to the underlying model) is, however, thinly documented: on consequential real-world tasks agents still fail the majority — Gemini 2.5 Pro completed only 30.3% of TheAgentCompany's 175 professional tasks (OpenHands scaffold, project leaderboard) — so the agency-specific harm magnitude is early and context-dependent rather than established at scale.","sources":["Kwa, West, Becker et al. 2025 (METR; arXiv:2503.14499, 'Measuring AI Ability to Complete Long Tasks')","UK AI Security Institute 2025 (Frontier AI Trends Report, Dec 2025)","Xu, Song, Zhou et al. 2024 (TheAgentCompany, arXiv:2412.14161); 30.3% figure per TheAgentCompany leaderboard (OpenHands)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact-evaluation evidence that agent-specific governance reduces agentic harm: the operative regimes — the EU GPAI Code of Practice (published July 2025, voluntary/non-binding), the Seoul Frontier AI Safety Commitments (2024, voluntary), and AISI agent evaluations — are 2024-25 vintage and have never been measured against an outcome. The scholarship itself has not settled the contested unit of regulation: Kolt (2025) argues for governing the agentic relationship via principal-agent and agency-law tools, while Chan, Ezell, Kaufmann et al. (2024) propose agent-specific visibility mechanisms (identifiers, real-time monitoring, activity logging) that remain proposal-stage and unevaluated — meaning the field has design proposals but, as with most frontier-AI rules, the evidence that any of them works is absent rather than merely thin.","sources":["Kolt 2025 ('Governing AI Agents', 101 Notre Dame L. Rev., forthcoming; arXiv:2501.07913)","Chan, Ezell, Kaufmann et al. 2024 ('Visibility into AI Agents', ACM FAccT 2024, pp. 958-973; DOI 10.1145/3630106.3658948)","EU AI Office 2025 (GPAI Code of Practice, July 2025); Seoul Frontier AI Safety Commitments 2024"]},"wikiUrl":"https://policywindow.org/wiki/agentic-systems-governance#social-science-evidence"},{"code":"ai_worker_displacement","label":"AI-Driven Worker Displacement","kind":"sector","empiricalConsensus":"emerging","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"AI-driven labour displacement is demonstrably real but localized rather than economy-wide as of 2025-2026. Causal microdata find measurable harm in directly exposed segments: a difference-in-differences study of the Upwork freelance market found that after ChatGPT's release, freelancers in more AI-exposed occupations (e.g. writing) saw ~2% fewer contracts and ~5% lower monthly earnings, with larger losses among previously high-skilled workers (Hui, Reshef & Zhou 2024). Effects concentrate in entry-level and highly-automatable roles while aggregate US employment and wages show little disruption through 2024-2025 — so macro-level harm remains genuinely contested even as targeted-segment harm is established; much deployment to date augments rather than substitutes, raising novice productivity ~34% in call-center work (Brynjolfsson, Li & Raymond 2025).","sources":["Hui, Reshef & Zhou 2024 ('The Short-Term Effects of Generative AI on Employment', Organization Science)","Brynjolfsson, Li & Raymond 2025 ('Generative AI at Work', Quarterly Journal of Economics 140(2):889)","Acemoglu 2024 ('The Simple Macroeconomics of AI', NBER WP 32487)","Autor 2024 ('Applying AI to Rebuild Middle Class Jobs', NBER WP 32140)"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"There are essentially no impact evaluations of governance specifically targeting AI-driven displacement; current responses (OECD/GPAI guidance, reskilling initiatives, safety-net proposals) are at the recommendation stage, so 'does AI-displacement policy work' is answered only by extrapolation from the broader displaced-worker literature. That analogue base is robust but shows modest, mixed results: Card, Kluve & Weber's (2018) meta-analysis of 200+ active-labour-market evaluations finds training has small/insignificant short-run effects that improve only over the medium-to-long run, US Trade Adjustment Assistance evaluations find largely neutral-to-negative earnings effects (Schochet et al. 2012), and the JTPA randomized evaluation found weak earnings effects for the dislocated-worker stream. Recent syntheses note retraining yields smaller gains precisely when workers move into high-AI-exposure occupations — so the evidence that standard tools reduce AI-displacement harm is thin and early.","sources":["Card, Kluve & Weber 2018 ('What Works? A Meta-Analysis of ... Active Labor Market Program Evaluations', JEEA 16(3):894)","Schochet et al. 2012 (Trade Adjustment Assistance Program impacts, Mathematica/USDOL)","Bloom et al. 1997 (National JTPA Study, Journal of Human Resources)","Brookings 2025 ('AI Labor Displacement and the Limits of Worker Retraining'); OECD 2023-2025 Employment Outlook"]},"wikiUrl":"https://policywindow.org/wiki/ai-worker-displacement#social-science-evidence"},{"code":"biometric_id","label":"Biometric Identification","kind":"capability","empiricalConsensus":"settled","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"Demographic accuracy disparities in facial recognition are robust and replicated. NIST's Face Recognition Vendor Test (189 algorithms, 18.27M images) found one-to-one false-positive rates for Asian and African-American faces elevated 10-100x over white males, with the highest one-to-many false positives for African-American women; Buolamwini & Gebru's Gender Shades found commercial gender-classification error up to 34.7% for darker-skinned women vs 0.8% for lighter-skinned men. Documented downstream harm includes at least 8-15 US wrongful arrests, nearly all of Black people. Honest caveat: magnitude is highly algorithm-dependent — the most accurate algorithms show small or statistically undetectable differentials — so the harm is real but not uniform across systems.","sources":["Grother, Ngan & Hanaoka 2019 (NISTIR 8280, FRVT Part 3: Demographic Effects)","Buolamwini & Gebru 2018 (Gender Shades, PMLR 81)","Hill 2020 / Williams v. City of Detroit (ACLU 2021)"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"Rigorous evidence that GOVERNANCE of biometric ID reduces the documented harms is sparse. The one quantitative impact evaluation of police facial-recognition policy (Johnson et al. 2024, difference-in-differences across 268 US cities) studies effects on violent crime — a crime-control outcome, not misidentification harm — from a single research group, and does not establish that any safeguard regime curbs wrongful identification. Direct evidence on procedural safeguards points the other way: in the known wrongful-arrest cases police are reported to have bypassed required corroboration/probable-cause standards, and the strongest documented enforcement levers are private-sector biometric-privacy laws — Illinois BIPA (e.g. Meta's $650M settlement) and the separate Texas CUBI law (a $1.4B Meta settlement) — which govern private actors, not the law-enforcement context where the arrests occur. No replicated study shows a specific regulatory regime measurably reduces demographic misidentification harm.","sources":["Johnson et al. 2024 (Cities, 'Police facial recognition applications and violent crime control in U.S. cities')","Harwell & Schaffer 2025 (Washington Post, 'Arrested by AI')","Illinois BIPA (Rosenbach v. Six Flags 2019; Meta $650M settlement 2021); Texas CUBI (Meta $1.4B settlement 2024)"]},"wikiUrl":"https://policywindow.org/wiki/biometric-id#social-science-evidence"},{"code":"catastrophic_risk","label":"Catastrophic & Existential Risk","kind":"capability","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"The catastrophic-uplift premise is genuinely contested: the empirical uplift studies that exist find current frontier models add little. RAND's red-team study found no statistically significant difference in the viability of bioweapon-attack plans produced with vs. without LLMs (Mouton, Lucas & Guest 2024), and OpenAI's 100-participant trial found GPT-4 gave at most a mild, non-significant accuracy uplift (mean +0.88 out of 10 for PhD experts, +0.25 for students; Patwardhan et al. 2024). Honest caveat: the harm is forward-looking, not yet observed — expert opinion on the catastrophic tail is sharply split (median AI researcher puts ~5% on extremely-bad/extinction outcomes, mean ~9-16% across differently-framed questions, n=2,778; Grace et al. 2024), and forecasters underestimated how fast risk-relevant capabilities (e.g. virology troubleshooting) actually arrived (Forecasting Research Institute 2025), so the relevant capabilities are a moving target rather than a settled magnitude.","sources":["Mouton, Lucas & Guest 2024 (RAND RR-A2977-2, Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study)","Patwardhan et al. 2024 (OpenAI, Building an Early Warning System for LLM-aided Biological Threat Creation)","Grace et al. 2024 (Thousands of AI Authors on the Future of AI, arXiv:2401.02843); Forecasting Research Institute 2025 (Forecasting LLM-enabled Biorisk and the Efficacy of Safeguards)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is essentially no impact evidence that catastrophic-risk governance reduces catastrophic risk, and structurally there cannot yet be: the harm is a low-probability civilisational tail event, so no controlled trial or before/after evaluation of a realised catastrophe is possible. The dominant instruments are recent, voluntary developer frameworks (Anthropic's Responsible Scaling Policy 2023; OpenAI's Preparedness Framework 2023) built on if-then capability thresholds the developers themselves describe as speculative and qualitative rather than validated risk thresholds. The closest evidence is adjacent and indirect: trained-in deceptive behaviours can persist through standard safety training (Hubinger et al. 2024) — a demonstration that current mitigation may be insufficient, not that any governance regime works — and Anthropic's documented loosening of earlier commitments (RSP 2025 dropped the original pledge to define higher-tier ASL evaluations before developing the corresponding models) illustrates that even the strongest voluntary regimes lack external enforcement or measured efficacy.","sources":["Anthropic 2023 (Responsible Scaling Policy); OpenAI 2023 (Preparedness Framework)","Hubinger et al. 2024 (Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv:2401.05566)","Hendrycks, Mazeika & Woodside 2023 (An Overview of Catastrophic AI Risks, arXiv:2306.12001)"]},"wikiUrl":"https://policywindow.org/wiki/catastrophic-risk#social-science-evidence"},{"code":"compute_export_controls","label":"Compute + Model-Weight Export Controls","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"That compute controls materially constrain China's frontier-AI hardware access is empirically real and measured: by mid-2025 the US hosted ~75% of catalogued AI-supercomputer performance versus China's ~15% (Pilz, Sanders, Rahman & Heim 2025), corroborated by the Federal Reserve's estimate of ~74% US vs ~14% China high-end AI compute share (Haag, FEDS Notes, Oct 2025), and US prosecutions document large diversion networks (e.g. the ~$160M Alan Hao Hsu / Hao Global H100/H200 case — the first 'AI diversion' conviction, guilty plea Oct 2025, SDTX). The honest caveat is that the magnitude of the binding constraint is genuinely contested: DeepSeek's V3/R1 reached near-frontier capability at an order of magnitude less reported compute via algorithmic efficiency, and analysts argue the controls have simultaneously accelerated Chinese state-backed indigenization, so whether the controls slow capability (the actual aim) versus merely shift its cost structure remains unsettled.","sources":["Pilz, Sanders, Rahman & Heim 2025 (Trends in AI Supercomputers, arXiv:2504.16026 / Epoch AI) — VERIFIED, gives US ~75% / China ~15%","Haag 2025 (FEDS Notes, 'The State of AI Competition in Advanced Economies', Federal Reserve, 6 Oct 2025) — VERIFIED, gives US 74% / China 14% / EU 4.8% of high-end AI compute","US v. Alan Hao Hsu / Hao Global 2025 (DOJ, SDTX; ~$160M H100/H200 diversion, guilty plea Oct 2025, first AI-diversion conviction) — VERIFIED"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that compute or model-weight export controls achieve their stated strategic aim of durably slowing frontier-AI capability diffusion to China — the regime is too recent, the counterfactual is unidentified, and the most ambitious instrument (the Jan 2025 BIS 'Framework for AI Diffusion', ECCN 4E091 covering model weights of closed models trained on >10^26 operations) was rescinded on 12-13 May 2025 before it ever took effect (its enforcement date was 15 May 2025), so the evidence that the rule works is itself missing. The closest analogue evidence base, the economic-sanctions evaluation literature, is sobering: Hufbauer, Schott, Elliott & Oegg (2007) coded roughly a third of their historical cases as 'successful' (their database covers ~170-200 cases since WWI; the disputed coding was ~40 of 115, ~34%), but Pape's reanalysis (1997/1998) argued the genuinely sanctions-attributable success rate was far lower (he recoded it to ~5 of 115, under 5%), and the broader literature finds efficacy decays as targets adapt and substitute — the precise dynamic export-control critics attribute to Chinese indigenization and smuggling. This is an analogue, not direct evidence on export controls.","sources":["Hufbauer, Schott, Elliott & Oegg 2007 (Economic Sanctions Reconsidered, 3rd ed., Peterson Institute for International Economics) — VERIFIED","Pape 1997/1998 (Why Economic Sanctions Do Not Work, International Security 22(2), 1997; Why Economic Sanctions Still Do Not Work, International Security 23(1), 1998) — VERIFIED","BIS 2025 (Framework for Artificial Intelligence Diffusion, Federal Register doc 2025-00636 / 90 FR, eff. 13 Jan 2025; ECCN 4E091 model-weight control; rescinded 12-13 May 2025 before its 15 May effective date) — VERIFIED"]},"wikiUrl":"https://policywindow.org/wiki/compute-export-controls#social-science-evidence"},{"code":"compute_reporting","label":"Compute-Threshold Reporting","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"Whether training-compute (FLOP) is a defensible proxy for governance-relevant capability is genuinely contested in the literature. The strongest empirical pressure against it is algorithmic efficiency: Ho, Besiroglu, Erdil et al. (2024) estimate the compute needed to reach a fixed language-model performance level has halved roughly every eight months (95% CI ~5-14 months, i.e. ~3x/year), so any static FLOP-to-capability mapping decays quickly; Hooker (2024) argues FLOP measures operations rather than end-performance, since techniques such as fine-tuning, retrieval, chain-of-thought and tool use can add large capability gains without proportional training compute, and Ord (2025) shows inference-time scaling further decouples deployed capability from training compute. Honest caveat: defenders (Heim & Koessler 2024; Pilz, Heim & Brown 2025) note compute remains the most quantifiable, externally verifiable, and ex-ante measurable correlate of frontier capability currently available, while themselves conceding it is an imperfect proxy that should not be used in isolation — the disagreement is about durability and precision, not whether any correlation exists.","sources":["Ho, Besiroglu, Erdil, Owen, Rahman, Guo, Atkinson, Thompson & Sevilla 2024, Algorithmic progress in language models, NeurIPS 2024 (arXiv:2403.05812; Epoch AI)","Hooker 2024, On the Limitations of Compute Thresholds as a Governance Strategy (arXiv:2407.05694)","Ord 2025, Inference Scaling Reshapes AI Governance (arXiv:2503.05705); Heim & Koessler 2024, Training Compute Thresholds: Features and Functions in AI Regulation (arXiv:2405.10799); Pilz, Heim & Brown 2025, Increased Compute Efficiency and the Diffusion of AI Capabilities (AAAI 2025; arXiv:2311.15377)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous evidence that compute-threshold reporting reduces harm or achieves its stated aim, because the regimes have not produced an evaluable record. The US 10^26-FLOP reporting obligation (Executive Order 14110, invoking the Defense Production Act) was revoked on 20 January 2025 (by EO 14148) before its recurring binding reporting rule was finalized — the implementing BIS notice of proposed rulemaking (Sept 2024) never took effect, so no durable reporting record materialized; and the EU AI Act's 10^25-FLOP systemic-risk obligations for general-purpose models only became applicable on 2 August 2025 (with transitional periods into 2027), so no outcome evaluation yet exists. Moreover the 10^25 figure is a rebuttable presumption sitting alongside qualitative high-impact criteria (Art. 51(1)(a) and (2), rebuttable under Art. 52(2)), not a validated risk cutoff. The closest analogue is the broader regulatory-disclosure-mandate literature (Fung, Graham & Weil 2007), which documents that transparency policies' effects on outcomes are highly heterogeneous and frequently ineffective or counterproductive absent enforcement and downstream use — implying that the reporting trigger working as intended is an open empirical question, not a documented result.","sources":["U.S. Executive Order 14110 (2023), Sec. 4.2 (10^26 FLOP, Defense Production Act); revoked by Executive Order 14148 (Jan 20, 2025)","EU AI Act, Reg. (EU) 2024/1689, Art. 51 (10^25 FLOP systemic-risk rebuttable presumption; applicable Aug 2, 2025)","Fung, Graham & Weil 2007, Full Disclosure: The Perils and Promise of Transparency (Cambridge University Press)"]},"wikiUrl":"https://policywindow.org/wiki/compute-reporting#social-science-evidence"},{"code":"criminal_justice","label":"AI in Criminal Justice","kind":"sector","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"Whether algorithmic risk assessment reproduces racial disparity is a genuine, partly mathematically irreducible dispute rather than merely an unresolved measurement question. ProPublica's analysis of COMPAS in Broward County found Black defendants who did not reoffend were nearly twice as likely to be flagged high-risk as comparable white defendants (44.9% vs 23.5% false-positive rate; Angwin et al. 2016), and Dressel & Farid (2018) showed COMPAS is no more accurate (65.2%) than untrained laypeople (67.0%); the developer's reanalysis (Flores, Bechtel & Lowenkamp 2016) found the same tool satisfies predictive parity and calibration across race. Honest caveat: Chouldechova (2017) proved both sides can be correct simultaneously — when recidivism base rates differ across groups, equal calibration and equal error rates cannot both hold, so the disagreement is partly definitional, not merely a data dispute to be settled.","sources":["Angwin, Larson, Mattu & Kirchner 2016 (ProPublica, 'Machine Bias')","Dressel & Farid 2018 (Science Advances 4:eaao5580)","Flores, Bechtel & Lowenkamp 2016 (Federal Probation 80(2):38); Chouldechova 2017 (Big Data 5(2):153)"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"Rigorous evidence that governing criminal-justice algorithms — mandating, auditing, or adopting risk tools — reduces the racial-disparity harm that motivates the rules is essentially absent. The leading real-world impact evaluation, Stevenson's (2018) study of Kentucky's mandatory pretrial risk-assessment law (>1M cases), found only a small increase in pretrial release that eroded as judges reverted to prior habits, with no reduction in racial disparities in pretrial detention. The closest analogue evaluations measure operational crime outcomes, not equity, and are largely null: Chicago's Strategic Subjects List had no effect on victimization (Saunders, Hunt & Hollywood 2016) and the only randomized predictive-policing trials tested crime reduction, not disparate impact (Mohler et al. 2015) — so the evidence that any governance regime measurably reduces algorithmic racial disparity is itself missing.","sources":["Stevenson 2018 (Minnesota Law Review 103:303)","Saunders, Hunt & Hollywood 2016 (Journal of Experimental Criminology 12(3):347)","Mohler et al. 2015 (JASA 110(512):1399)"]},"wikiUrl":"https://policywindow.org/wiki/criminal-justice#social-science-evidence"},{"code":"deepfakes","label":"Deepfakes / Synthetic Content","kind":"capability","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"The flagship harm — non-consensual sexual deepfakes — is empirically real and sharply gendered: content audits find ~96-98% of deepfake videos online are non-consensual pornography overwhelmingly depicting women, and a pre-registered 10-country survey (>16,000 people) found 2.2% reporting victimization and 1.8% perpetration of synthetic intimate imagery, with documented mental-health, career, and participation harms. By contrast, the parallel claim that political/informational deepfakes UNIQUELY deceive is contested-to-refuted: experiments find deepfakes about as (not more) credible than equivalent text/audio fakes, and a 56-paper meta-analysis (k=137, N=86,155) puts unaided human detection near chance — implying a detection problem more than an exceptional-persuasion one.","sources":["Umbach, Henry, Beard & Berryessa 2024 (CHI '24, 'Non-Consensual Synthetic Intimate Imagery ... in 10 Countries')","Diel et al. 2024 (Computers in Human Behavior Reports 16:100538, deepfake-detection meta-analysis of 56 papers)","Barari, Lucas & Munger 2025 (Journal of Politics 87(2), 'Political Deepfakes Are as Credible as Other Fake Media')","Flynn et al. 2022 (British Journal of Criminology, multi-country image-based sexual abuse study)"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"Direct impact evidence that deepfake governance reduces the targeted harm is sparse and, where it exists, discouraging: the one quasi-experimental evaluation (Cuevas & Horta Ribeiro 2025, synthetic-control across three platforms) found the U.S. TAKE IT DOWN Act's passage plus the MrDeepfakes shutdown did NOT suppress synthetic non-consensual imagery — posting rose above counterfactual baselines and displaced elsewhere. Technical enforcement is likewise unreliable: detectors fail to generalize to unseen generators (notably diffusion models) and are vulnerable to adversarial evasion, with in-the-wild accuracy well below benchmark figures. No rigorous evaluation yet shows a deepfake-specific law, takedown mandate, or watermarking scheme producing a sustained reduction in prevalence or harm.","sources":["Cuevas & Horta Ribeiro 2025 ('Deepfake Pornography is Resilient to Regulatory and Platform Shocks', arXiv:2602.02754)","'Adversarial Reality for Evading Deepfake Image Detectors' (ICCVW 2025)","TAKE IT DOWN Act, S.146 / Pub. L. 119-12 (2025); CRS Legal Sidebar LSB11314"]},"wikiUrl":"https://policywindow.org/wiki/deepfakes#social-science-evidence"},{"code":"development_rights_framing","label":"Development-Rights Framings","kind":"political_frame","empiricalConsensus":"emerging","realHarmUnprovenGovernance":false,"problemReality":{"epistemicStatus":"thin","finding":"Development-rights framing is a normative/doctrinal frame, so its empirical status splits: the underlying North-South asymmetry it responds to is real and documented, but the claim that a development-rights diagnosis is the correct one is contested doctrine, not a settled finding. The strongest empirical anchor is the exploitative-data-labour evidence — Miceli & Posada's (2022) multi-method qualitative study of Latin American annotation work (Foucauldian dispositif analysis of 210 instruction documents, 55 interviews, plus participant observation) found workers paid cents-per-task with strict surveillance and whose worldviews are subordinated to requesters' — which substantiates the extraction the frame names, building on the data-colonialism thesis (Couldry & Mejias 2019), and extended by comparative political-economy work on AI annotation 'data empires' (Wu, Muldoon & Xia 2025). Honest caveat: whether 'digital self-determination' or 'Global-South sovereignty' is the right operational response (and whether it conflicts with the EU AIA's rights-based design) is a conceptual/legal question with essentially no empirical evidence base — the frame is established as a critique, thin as a tested governance prescription.","sources":["Miceli & Posada 2022, 'The Data-Production Dispositif' (Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Art. 460:1-37)","Couldry & Mejias 2019, 'Data Colonialism' (Television & New Media 20(4):336-349)","Wu, Muldoon & Xia 2025, 'Global data empires' (Big Data & Society 12(2))"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that development-rights / digital-self-determination / sovereignty governance achieves its stated developmental or self-determination aims — the evidence that the frame 'works' as policy is itself missing, largely because the frame is recent, heterogeneous, and rarely instantiated in a single measurable instrument. The closest empirical literature studies one common operational proxy (data localization) and measures economic cost rather than the frame's goals: Ferracane, Kren & van der Marel's (2020) firm/industry productivity analysis finds data-policy restrictiveness associated with lower TFP in data-intensive downstream sectors, Ferracane & van der Marel's (2021) gravity analysis finds data restrictions inhibit trade in digital services, and Bauer, Lee-Makiyama, van der Marel & Verschelde's (2014) GTAP general-equilibrium estimates project GDP losses from localization across seven jurisdictions including Brazil and India. None tests whether sovereignty framing reduces extractive asymmetry or advances local AI capability — so claims on both the benefit and cost sides rest on weak or indirect evidence.","sources":["Ferracane, Kren & van der Marel 2020, 'Do data policy restrictions impact the productivity performance of firms and industries?' (Review of International Economics 28(3):676-722)","Ferracane & van der Marel 2021, 'Do data policy restrictions inhibit trade in services?' (Review of World Economics 157(4):727-776)","Bauer, Lee-Makiyama, van der Marel & Verschelde 2014, 'The Costs of Data Localisation: Friendly Fire on Economic Recovery' (ECIPE Occasional Paper 3/2014)"]},"wikiUrl":"https://policywindow.org/wiki/development-rights-framing#social-science-evidence"},{"code":"education","label":"AI in Education","kind":"sector","empiricalConsensus":"settled","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"The documented harms of educational AI are empirically real and, for proctoring, replicated: a controlled audit of a proctoring tool used by at least ~1,500 institutions found significantly higher facial-detection failure (the trigger for 'suspicious' flags) for darker-skinned and female test-takers (Yoder-Himes et al. 2022), and a technical audit of 164 government-endorsed pandemic learning products found 89% engaged in data practices that risk or infringe children's rights, with most monitoring happening without the child's knowledge or consent (Human Rights Watch 2022). Honest caveat: the benefit side is genuine but highly sensitive to how outcomes are measured rather than uniform — Kulik & Fletcher's meta-analysis of 50 intelligent-tutoring evaluations found an overall median effect of 0.66 SD, but the average effect was 0.73 SD on locally-developed tests versus only 0.13 SD on standardized tests, so much of AI education's apparent value depends on the outcome measure used.","sources":["Yoder-Himes et al. 2022, 'Racial, skin tone, and sex disparities in automated proctoring software', Frontiers in Education 7:881449","Human Rights Watch 2022, 'How Dare They Peep into My Private Life?' (164 EdTech products endorsed by 49 governments; 89% risked/infringed children's rights)","Kulik & Fletcher 2016, 'Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review', Review of Educational Research 86(1):42-78"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"There are essentially no rigorous impact evaluations showing that purpose-built governance of educational AI reduces the documented harms. The student-specific regime — California's SOPIPA (SB 1177, 2014, a model that more than 20 states adopted and ~33 considered) and the FTC's May 2022 COPPA ed-tech policy statement (which the agency itself said did not change existing requirements) — has near-zero documented enforcement and no published before/after evaluation of whether it changed vendor data practices or bias outcomes. The only documented remedies came not from education-specific rules but from generic legal levers: a $6.25M biometric-privacy class settlement under Illinois BIPA (Veiga v. Respondus, 2023) and a constitutional ruling that proctoring room-scans are an unreasonable search (Ogletree v. Cleveland State University, N.D. Ohio 2022, Calabrese J.) — neither of which is a replicable evaluation, and both reach private/state actors rather than the underlying demographic-bias harm.","sources":["California SOPIPA (SB 1177, 2014); FTC Policy Statement on Education Technology and COPPA (adopted May 19, 2022)","Veiga v. Respondus, Inc. ($6.25M BIPA class settlement, 2023; covers Illinois Respondus Monitor users Nov. 2015–June 2023)","Ogletree v. Cleveland State University (N.D. Ohio 2022, Calabrese J., room-scan Fourth Amendment ruling)"]},"wikiUrl":"https://policywindow.org/wiki/education#social-science-evidence"},{"code":"employment","label":"AI in Employment","kind":"sector","empiricalConsensus":"settled","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"Discrimination and adverse outcomes in employment decisions are empirically well-established, and AI systems demonstrably reproduce them. The foundational field-experiment literature shows robust human baseline discrimination (Bertrand & Mullainathan 2004 found White-sounding names received 50% more callbacks), and AI-specific audits confirm the pattern: Amazon scrapped a recruiting tool that penalized resumes containing 'women's' (Dastin 2018), and a controlled resume-screening audit of language-model retrieval found systems favored White-associated names ~85% of the time and never preferred Black male-associated over White male-associated names (Wilson & Caliskan 2024). On the monitoring side, a meta-analysis (k=94, N≈23,461) found electronic performance monitoring reliably raises worker stress with no evidence of improved performance (Ravid et al. 2023). Honest caveat: measured disparities are highly model-, prompt-, and context-dependent, and most evidence comes from controlled audits and one firm's internal test rather than measured outcomes in live, at-scale hiring pipelines.","sources":["Bertrand & Mullainathan 2004 (American Economic Review 94(4):991-1013)","Wilson & Caliskan 2024 (AAAI/ACM AIES; 'Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval')","Dastin 2018 (Reuters, 'Amazon scraps secret AI recruiting tool that showed bias against women'); Ravid, White, Tomczak & Behrend 2023 (Personnel Psychology 76:5-40)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous evidence that governing AI in employment reduces the documented harms; the central evaluated regime appears to fail at the compliance stage before any impact on bias can occur. NYC Local Law 144 — the first jurisdiction worldwide to mandate independent bias audits and public posting for automated employment decision tools — was directly studied across 391 employers and found to produce 'null compliance': the law's discretion makes it impossible to tell whether firms comply, with very few posting the required audits (Wright et al. 2024). Parallel qualitative work shows the audits themselves are undermined by missing demographic data, opaque aggregation, and 'test data' that does not reflect real use (Groves et al. 2024). No study links any AI-employment rule to a measured reduction in discriminatory hiring outcomes — the evidence that the rule works is itself missing, largely because mandated transparency artifacts (audit reports) are sparse, non-standardized, and unenforced.","sources":["Wright, Muenster, Vecchione, Metcalf & Matias et al. 2024 ('Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability', ACM FAccT '24)","Groves, Metcalf, Kennedy, Vecchione & Strait 2024 ('Auditing Work: Exploring the New York City algorithmic bias audit regime', ACM FAccT '24)","Ravid, White, Tomczak & Behrend 2023 (Personnel Psychology 76:5-40, on monitoring outcomes as the closest analogue evaluation evidence)"]},"wikiUrl":"https://policywindow.org/wiki/employment#social-science-evidence"},{"code":"environmental_impact_of_training","label":"Environmental Impact of AI Training","kind":"procedural","empiricalConsensus":"emerging","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"The resource demands of AI compute are empirically documented at the model level: Strubell et al. (2019) quantified large-NLP training energy/carbon, Luccioni et al. (2023) estimated BLOOM's training at ~24.7 tCO2eq (dynamic power) rising to ~50.5 tCO2eq with manufacturing and deployment, Li et al. (2023) estimated GPT-3-scale training in US datacenters can evaporate on the order of hundreds of thousands of litres of freshwater (their central figure ~700,000 L), and Luccioni, Jernite & Strubell (2024) showed generative inference is markedly more energy-intensive per query than task-specific models; at the macro scale the IEA (2024) and de Vries (2023) document rapidly rising datacenter electricity demand. Honest caveat: absolute estimates vary by up to orders of magnitude with grid carbon intensity, hardware, utilisation and accounting boundaries, and cleanly attributing the AI-specific increment (versus general datacenter and crypto growth) remains genuinely contested — the IEA itself bundles AI with datacenters and crypto — so the existence of the footprint is established while its magnitude and trajectory are not.","sources":["Strubell, Ganesh & McCallum 2019 (ACL Anthology P19-1355; 'Energy and Policy Considerations for Deep Learning in NLP')","Luccioni, Viguier & Ligozat 2023 (JMLR 24; BLOOM 176B carbon footprint, 24.7/50.5 tCO2eq; arXiv:2211.02001)","Li, Yang, Islam & Ren 2023 (arXiv:2304.03271, 'Making AI Less Thirsty', later Comm. ACM 2025); Luccioni, Jernite & Strubell 2024 (ACM FAccT '24, 'Power Hungry Processing', DOI 10.1145/3630106.3658542)","de Vries 2023 (Joule 7(10):2191-2194, DOI 10.1016/j.joule.2023.09.004); IEA 2024 (Electricity 2024)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact evaluation showing that any AI-specific environmental-governance instrument reduces energy, water or carbon use, because every named instrument is voluntary or non-binding and very recent: EU AI Act Art. 95 codes of conduct are explicitly optional with no sanctions, and NIST AI 600-1 and the G7 Hiroshima Code are guidance, not enforceable caps. The closest analogue evaluation literature is divided in a way that disfavours the voluntary form chosen here: rigorous reviews find voluntary environmental programs generally fail to produce significant abatement beyond business-as-usual (Koehler 2007; Morgenstern & Pizer 2007), whereas the one form with credible positive evidence is mandatory disclosure (Downar et al. 2021 found a UK carbon-reporting mandate cut emissions ~8% versus a control group) which the AI instruments do not yet impose, leaving the proposition that AI environmental governance works essentially untested.","sources":["EU AI Act Art. 95 / Recital 142 (Reg. (EU) 2024/1689); NIST AI 600-1 (2024, GenAI Profile); G7 Hiroshima Process International Code of Conduct (30 Oct 2023)","Koehler 2007 (Policy Studies Journal 35(4):689-722); Morgenstern & Pizer (eds.) 2007 (Reality Check, RFF Press)","Downar, Ernstberger, Reichelstein, Schwenen & Zaklan 2021 (Review of Accounting Studies 26(3):1137-1175)"]},"wikiUrl":"https://policywindow.org/wiki/environmental-impact-of-training#social-science-evidence"},{"code":"foundation_models","label":"Foundation Models / GPAI","kind":"capability","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"Whether the foundation-model category maps to a coherent capability/risk tier is genuinely contested. The original case rests on scale-driven 'emergent abilities' that appear unpredictably above a size threshold (Wei et al. 2022; Ganguli et al. 2022 documented capabilities that are smoothly predictable in aggregate loss yet locally surprising), but Schaeffer, Miranda & Koyejo (2023, a NeurIPS Outstanding Paper) showed many 'emergent' jumps are artefacts of discontinuous metrics and dissolve under linear/continuous scoring — implying capability scales more smoothly than a sharp tier would suggest. Honest caveat: this is a live empirical disagreement about measurement, not a settled finding either way, and compute (the regulatory proxy) is an imperfect stand-in for capability or risk regardless of which side is right.","sources":["Wei et al. 2022 (Emergent Abilities of Large Language Models, TMLR; arXiv:2206.07682)","Schaeffer, Miranda & Koyejo 2023 (Are Emergent Abilities of Large Language Models a Mirage?, NeurIPS 2023, Outstanding Paper; arXiv:2304.15004)","Ganguli et al. 2022 (Predictability and Surprise in Large Generative Models, ACM FAccT; DOI 10.1145/3531146.3533229)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact evaluation showing that GPAI/foundation-model governance reduces harm — the rules are too new (EU AI Act GPAI obligations and the 10^25-FLOP systemic-risk presumption only began binding on 2 August 2025) and the central regulatory lever is itself contested: Hooker (2024) argues compute thresholds are a shortsighted proxy because compute does not reliably track capability or risk, and the thresholds already diverge across jurisdictions (EU 10^25 vs. the now-rescinded US EO 14110's 10^26 operations, rescinded 20 January 2025). The mandated mitigation methods also lack validated efficacy: model evaluation and red-teaming face well-documented coverage limits and an 'audit gap' in the survey/position literature (behavioural testing cannot establish the absence of untested failure modes), and adversarial red-teaming repeatedly defeats deployed safeguards — the UK AI Safety Institute reports finding universal jailbreaks for every frontier system it has tested, and a large public agent-injection competition elicited policy violations across all 22 frontier models tested from ~1.8M attacks (Zou et al. 2025). Even compliant evaluation therefore cannot yet certify the safety the rules demand. (Caveat: this is an absence-of-evidence claim — no efficacy study has been done — not evidence the rules are ineffective.)","sources":["Hooker 2024 (On the Limitations of Compute Thresholds as a Governance Strategy, arXiv:2407.05694)","EU AI Act Arts. 51 & 55 (GPAI systemic-risk presumption, 10^25 FLOP; binding 2 Aug 2025); US EO 14110 (10^26-operation reporting threshold, rescinded 20 Jan 2025 by EO 14148)","Zou et al. 2025 (Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition / Gray Swan Arena, arXiv:2507.20526 — 22 frontier agents, ~1.8M attacks); UK AI Safety/Security Institute, Frontier AI Trends Report (universal jailbreaks for every system tested); METR, Common Elements of Frontier AI Safety Policies (2024)"]},"wikiUrl":"https://policywindow.org/wiki/foundation-models#social-science-evidence"},{"code":"healthcare","label":"AI in Healthcare","kind":"sector","empiricalConsensus":"settled","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"Both the benefit and the harm of clinical AI are empirically real and well-documented, but outcomes are highly deployment-dependent. Rigorous prospective studies show genuine clinical value in narrow tasks — the MASAI RCT (>100,000 women) found AI-supported mammography detected ~20% more cancers (6.1 vs 5.1 per 1000 screened) at comparable recall rates (Lang et al. 2023, Lancet Oncology), and IDx-DR's pivotal trial achieved 87.2% sensitivity / 90.7% specificity for diabetic retinopathy (Abramoff et al. 2018, npj Digital Medicine) — yet widely deployed models can fail or harm: the Epic Sepsis Model, live at hundreds of US hospitals, scored AUC 0.63 with 33% sensitivity on external validation (Wong et al. 2021, JAMA Internal Medicine), and a population-health algorithm covering ~200M people understated Black patients' illness because it predicted cost not need (Obermeyer et al. 2019, Science). Honest caveat: there is no single 'AI in healthcare' effect — performance ranges from life-saving to dangerous depending on task, calibration, and whether the model was prospectively validated.","sources":["Lang K, Josefsson V, Larsson A-M, et al. 2023 (Lancet Oncology 24(8):936-944, MASAI trial clinical safety analysis; AI-supported screening detected 6.1 vs 5.1 cancers per 1000, ~20% higher, similar recall rates)","Abramoff MD, Lavin PT, Birch M, Shah N, Folk JC. 2018 (npj Digital Medicine 1:39, IDx-DR pivotal trial; 87.2% sensitivity / 90.7% specificity)","Wong A, Otles E, Donnelly JP, et al. 2021 (JAMA Internal Medicine 181(8):1065-1070, Epic Sepsis Model external validation; AUC 0.63, 33% sensitivity); Obermeyer Z, Powers B, Vogeli C, Mullainathan S. 2019 (Science 366(6464):447-453, racial bias from cost-as-proxy)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is essentially no impact-evaluation evidence that the prevailing governance regime for medical AI — FDA authorization, predominantly via the 510(k) substantial-equivalence pathway — measurably reduces patient harm or improves outcomes. Analyses of authorized AI devices find that clinical validation is frequently absent or non-prospective (of 521 FDA-authorized AI devices, ~43% had no published clinical-validation data and only ~28% were prospectively validated; Chouffani El Fassi & Henderson et al. 2024) and that demographic performance is almost never reported (race/ethnicity in 3.6%, and only 9.0% of 692 510(k)/cleared AI devices carried a prospective post-market-surveillance study; Muralidharan et al. 2024). Earlier analysis of 130 cleared devices likewise found 97% were evaluated only retrospectively (Wu et al. 2021). The closest analogue evidence on the pathway itself is discouraging: the Institute of Medicine (2011) concluded the 510(k) process was not designed to assess safety and effectiveness — i.e., no direct study establishes that the rule, as written, prevents the harms it targets. Caveat: this is an absence of impact evaluation plus reporting-gap and design-critique evidence, not a study showing the regime fails to reduce harm.","sources":["Chouffani El Fassi S, Abdullah A, Fang Y, ... Henderson GE, et al. 2024 (Nature Medicine, 'Not all AI health tools with regulatory authorization are clinically validated', s41591-024-03203-3; 521 devices, ~43% no clinical validation, ~28% prospectively validated)","Muralidharan V, Adewale BA, Huang CJ, et al. 2024 (npj Digital Medicine 7:273, scoping review of reporting gaps in 692 FDA-approved AI medical devices; race/ethnicity 3.6%, prospective post-market surveillance 9.0%)","Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. 2021 (Nature Medicine 27:582-584, analysis of 130 FDA approvals; 97% retrospective-only evaluation); Institute of Medicine 2011 (Medical Devices and the Public's Health: The FDA 510(k) Clearance Process at 35 Years)"]},"wikiUrl":"https://policywindow.org/wiki/healthcare#social-science-evidence"},{"code":"international_coordination","label":"International Coordination","kind":"meta","empiricalConsensus":"emerging","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"The DESCRIPTIVE premise is well-established: IR scholarship now treats global AI governance as a fragmented 'regime complex' of partially overlapping G7/G20/OECD/GPAI/UN/standards-body arrangements with no central hierarchy (Tallberg et al. 2023 — verified verbatim: 'the emerging governance architecture for AI can be described as a regime complex'; Cihon, Maas & Kemp 2020). But the implied HARM — that forum-shopping and regulatory arbitrage cause a measurable race-to-the-bottom or relocate AI development to lax jurisdictions — is largely theorized/anticipated rather than empirically demonstrated for AI; Tallberg et al. explicitly flag forum-shopping as a dynamic whose presence in the AI regime complex is an open empirical question ('Establishing whether these patterns and dynamics are key features also of the AI regime complex stand out as important priorities in future research'). Honest caveat: the strongest empirical arbitrage evidence comes from analogue footloose digital markets (e.g., ICO reallocation after US securities enforcement) — itself a mixed/contested literature — not from AI firms, so the magnitude of coordination-failure harm in AI specifically remains contested and under-measured.","sources":["Tallberg, Erman, Furendal, Geith, Klamberg & Lundgren 2023 (International Studies Review 25(3): viad040)","Cihon, Maas & Kemp 2020 (Should AI Governance be Centralised?, AIES '20: 228-234)","Lancieri, Edelson & Bechtold 2025 (AI Regulation: Competition, Arbitrage & Regulatory Capture, Theoretical Inquiries in Law 26(1): 239-262)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There are essentially no impact evaluations showing that the negotiated-coordination mode (AI Safety Institute network MoUs, forum-shifting, multilateral declarations) actually produces regulatory convergence or reduces arbitrage — the AISI Network began only as a statement of intent at the Seoul Summit (Seoul Statement of Intent, 21 May 2024) and held its first operational meeting in November 2024, with no defined metrics or outcome studies, so these soft-law instruments are too new to have measurable effects. The closest analogue evidence is mixed and works through DIFFERENT mechanisms than this topic describes: Bradford's Brussels Effect documents de-facto convergence driven by market access rather than negotiated coordination, and the FATF transgovernmental-network literature shows peer-review mutual evaluation can drive AML convergence — but neither evaluates voluntary AI MoU networks, and FATF's effects come with well-documented unintended consequences (de-risking, financial exclusion). The plain finding: the evidence that AI-governance coordination 'works' is itself missing.","sources":["Bradford 2020 (The Brussels Effect: How the European Union Rules the World, Oxford University Press)","Nance 2018 (The regime that FATF built: an introduction to the Financial Action Task Force, Crime, Law and Social Change 69(2): 109-129; cf. Slaughter 2004, A New World Order, Princeton University Press)","International Network of AI Safety Institutes — Seoul Statement of Intent toward International Cooperation on AI Safety Science (21 May 2024; network's first meeting San Francisco, Nov 2024)"]},"wikiUrl":"https://policywindow.org/wiki/international-coordination#social-science-evidence"},{"code":"national_security_carveouts","label":"National Security Carveouts in AI Regulation","kind":"meta","empiricalConsensus":"settled","realHarmUnprovenGovernance":false,"problemReality":{"epistemicStatus":"thin","finding":"That civilian AI-governance instruments carve out national-security uses is black-letter and undisputed (EU AIA Art. 2(3); CoE Framework Convention Art. 3(2) on national-security activities, distinct from Art. 3(4) on national defence; US NSM-25 (Oct. 2024) as the national-security-track instrument fulfilling §4.8 of EO 14110); civil-society legal analysis argues a blanket exclusion is harder to square with a necessity-and-proportionality approach than a qualified one (Korff/ECNL 2022; Vogiatzoglou 2024). But whether the carveout itself produces concrete unredressed harm is empirically under-observed almost by construction — the secrecy it confers suppresses the very evidence needed to measure it. The closest analogue, national-security deference in the courts, shows the mechanism is real (the FISC granted all but eleven of 33,900 applications 1979-2012, a 99.97% approval rate; Sinnar 2022 documents downstream harms to securitized communities), yet Clarke (2014) shows that lopsided ex parte approval rates alone do not prove rubber-stamping, because rational case selection and pre-vetting produce similar rates in ordinary Title III wiretaps (99.93%) and delayed-notice warrants (99.6-99.8%) — so the magnitude of harm attributable to the carveout, as opposed to the legitimate secrecy of the domain, remains genuinely contested.","sources":["Korff 2022 (ECNL Opinion on the implications of the exclusion of national security from AI legislation, Oct. 2022)","Sinnar 2022 (Harvard Law Review Forum 136:59, 'A Label Covering a \"Multitude of Sins\": The Harm of National Security Deference')","Clarke 2014 (Stanford Law Review Online 66:125, 'Is the Foreign Intelligence Surveillance Court Really a Rubber Stamp?'); EPIC FISC statistics 1979-2012"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact evaluation showing that any specific design of the national-security carveout — categorical exclusion versus parallel governance track versus civilian-compliance-with-override — measurably improves oversight or reduces harm relative to the alternatives; the question is argued doctrinally (Vogiatzoglou 2024; Korff/ECNL 2022) but has never been tested empirically. The closest analogue evaluation literature is on the parallel-track model already in use for intelligence surveillance (the FISC / FISA oversight regime), and even there the evidence that the mechanism delivers effective scrutiny is itself contested rather than established (Clarke 2014; Sinnar 2022). No direct evaluation exists because the carveouts are recent (EU AIA 2024, CoE Framework Convention 2024, US NSM-25 2024), enforcement actions are by design non-public, and private parties typically lack standing to challenge a specific exempt deployment — the structural features that make the harm hard to observe also make the governance impossible to evaluate.","sources":["Vogiatzoglou 2024 (Verfassungsblog, 'The AI Act National Security Exception: room for manoeuvres?', 9 Dec. 2024)","Korff 2022 (ECNL Opinion, exclusion of national security from AI legislation)","Clarke 2014 (Stanford Law Review Online 66:125); Sinnar 2022 (Harvard Law Review Forum 136:59)"]},"wikiUrl":"https://policywindow.org/wiki/national-security-carveouts#social-science-evidence"},{"code":"open_weight_release","label":"Open-Weight Frontier Release","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"The empirical picture splits into two well-separated questions. (1) The MECHANISM that distinguishes open-weight release — that safety guardrails can be cheaply and irreversibly stripped once weights are public — is established: Qi et al. (2024) removed GPT-3.5 Turbo safety alignment by fine-tuning on only ~10 adversarially designed examples for under $0.20 (and the attack generalizes to Llama-2), and even purpose-built tamper-resistant safeguards (Tamirisa et al. 2025, TAR) were subsequently shown to be defeatable by adaptive fine-tuning (Qi et al. 2024, durability critique). (2) Whether this mechanism produces real-world CATASTROPHIC uplift is genuinely contested and, for the headline biosecurity case, currently unsupported: RAND's red-team study found no statistically significant difference in the viability of bioweapon attack plans produced with versus without LLM assistance (Mouton, Lucas & Guest 2024), and OpenAI's 100-participant trial found at most mild uplift over an internet baseline (Patwardhan et al. 2024). Honest caveat: these null/mild results are time-stamped to 2023-2024 frontier capability and to biothreats specifically; the marginal-risk framework (Kapoor, Bommasani et al. 2024) concludes the evidence base is too thin to characterize marginal risk across most misuse vectors, so 'no measured harm yet' is not 'no harm.'","sources":["Kapoor, Bommasani, Klyman, Longpre et al. 2024, 'Position: On the Societal Impact of Open Foundation Models', PMLR 235 / ICML 2024 (arXiv 2403.07918)","Mouton, Lucas & Guest 2024, RAND RR-A2977-2, 'The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study'","Qi, Zeng, Xie, Chen, Jia, Mittal & Henderson 2024, 'Fine-tuning Aligned Language Models Compromises Safety', ICLR 2024 (arXiv 2310.03693); Tamirisa et al. 2025, 'Tamper-Resistant Safeguards for Open-Weight LLMs', ICLR 2025 (arXiv 2408.00761); Qi, Wei, Carlini, Huang, Xie, He, Jagielski, Nasr, Mittal & Henderson 2024, 'On Evaluating the Durability of Safeguards for Open-Weight LLMs' (arXiv 2412.07097); Patwardhan et al. 2024, 'Building an early warning system for LLM-aided biological threat creation', OpenAI"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact evaluation showing that any specific weight-release governance regime reduces downstream harm, because no binding regime has been implemented and measured: California SB-1047's release-conditioning framework was vetoed in September 2024, and the EU AI Act's open-source carve-outs (Recital 102, Art. 53(2)) exempt most open-weight models (those below the systemic-risk compute threshold) from the documentation obligations that would generate evaluable conduct. The structural obstacle is also documented: Kapoor, Bommasani et al. (2024) characterize open-weight release as effectively irreversible and poorly monitorable once weights are public, so post-release governance has little to act on. The closest analogue evidence — technology export controls — is mixed and points to circumvention: commentators argue blanket export controls on freely copyable open-source models cannot work (Just Security 2024), and independent analyses of the post-2022 semiconductor controls document displacement to less-regulated channels (smuggling, threshold-tuned chip variants, cloud access) rather than disappearance of activity (e.g., CSIS, FPRI 2024), suggesting recipient-restriction regimes face the same leakage problem for weights. (Caveat: this is analogical, not direct evidence about weight-release governance, which remains unmeasured.)","sources":["Kapoor, Bommasani, Klyman, Longpre et al. 2024, 'Position: On the Societal Impact of Open Foundation Models', PMLR 235 (arXiv 2403.07918)","California SB-1047 (2024, vetoed by Gov. Newsom 29 Sep 2024); EU AI Act Regulation (EU) 2024/1689, Recital 102 & Art. 53(2) open-source exemptions","Just Security 2024, 'Export Controls on Open-Source Models Will Not Win the AI Race'; CSIS, 'The Limits of Chip Export Controls in Meeting the China Challenge' and FPRI 2024, 'Breaking the Circuit: US-China Semiconductor Controls' (export-control circumvention analogue)"]},"wikiUrl":"https://policywindow.org/wiki/open-weight-release#social-science-evidence"},{"code":"redress","label":"Individual Redress","kind":"procedural","empiricalConsensus":"settled","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"established","finding":"The premise behind redress — that affected people lack meaningful recourse against automated decisions — is real, but the flagship instrument is weaker than commonly assumed. Wachter, Mittelstadt & Floridi (2017) show GDPR creates only a limited 'right to be informed,' not a binding 'right to explanation' of specific decisions; and controlled work finds the explanations actually delivered do not measurably improve lay decision accuracy over showing the bare AI prediction (Alufaisan et al. 2021; and a 2022 meta-analysis by Schemmer et al. — screening 393 articles down to 9 in the final analysis — reports 'no effect of explanations on users' performance compared to sole AI predictions,' even though XAI overall had a positive effect). Honest caveat: the legitimacy/dignity value of being heard is empirically well established in the procedural-justice tradition even where outcome accuracy is unchanged, so 'redress fails' depends on which aim is measured.","sources":["Wachter, Mittelstadt & Floridi 2017 (International Data Privacy Law 7(2):76)","Alufaisan, Marusich, Bakdash, Zhou & Kantarcioglu 2021 (Proceedings of the AAAI Conference on AI 35(8):6618)","Schemmer, Hemmer, Nitsche, Kühl & Vössing 2022 (AAAI/ACM AIES '22, meta-analysis)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that mandated redress mechanisms (right-to-explanation, appeal, human-in-the-loop review) actually reduce erroneous or unfair automated decisions — the evidence that the rule works is itself missing. The closest experimental analogues are discouraging: explanations increase humans' acceptance of AI recommendations regardless of correctness (Bansal et al. 2021), and algorithm-in-the-loop oversight can introduce racial disparities and exhibit automation bias rather than reliably catching model errors (Green & Chen 2019). The procedural-justice literature (Tyler 1990; Lind & Tyler 1988) robustly supports a legitimacy and compliance benefit of fair process, but it measures perceived fairness, not reduction of the substantive decision harm redress is meant to cure.","sources":["Bansal, Wu, Zhou, Fok, Nushi, Kamar, Ribeiro & Weld 2021 (CHI '21)","Green & Chen 2019 (Disparate Interactions, ACM FAT* '19)","Tyler 1990 (Why People Obey the Law, Yale Univ. Press); Lind & Tyler 1988 (The Social Psychology of Procedural Justice, Plenum Press)"]},"wikiUrl":"https://policywindow.org/wiki/redress#social-science-evidence"},{"code":"sovereign_ai","label":"Sovereign AI Doctrine","kind":"political_frame","empiricalConsensus":"emerging","realHarmUnprovenGovernance":false,"problemReality":{"epistemicStatus":"thin","finding":"Sovereign-AI doctrine is post-2023 and largely aspirational, so its core empirical premise — that frontier model deployment can be meaningfully bound to a national jurisdiction — is only just beginning to be tested. What IS measurable is the underlying compute geography the doctrine reacts to: an audit of 775 non-U.S. data-center projects estimates U.S. companies operate ~48% of them when weighted by investment value (a proxy for compute capacity, and explicitly an initial public-data approximation), implying 'in-territory' hardware is frequently still subject to foreign corporate/legal control (Richardson et al. 2025). Honest caveat: there is no peer-reviewed evidence base establishing whether jurisdiction-bound frontier deployment is technically feasible at scale — the descriptive dependency (foreign operation of locally-sited hardware) is documented, but the doctrine's central feasibility claim is thin and early.","sources":["Richardson et al. 2025 (arXiv:2508.00932, 'How Sovereign Is Sovereign Compute? A Review of 775 Non-U.S. Data Centers')","Gupta, Walker & Reddie 2024 (arXiv:2411.14425, 'Whack-a-Chip: The Futility of Hardware-Centric Export Controls', UC Berkeley Risk & Security Lab)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that sovereign-AI governance achieves its stated aim of secure, contained national AI capability. The closest direct levers have measurable but mostly adverse or contested evidence: ex-ante simulations of the closest analogue — data-localization mandates — project GDP losses (EU GDP −0.4% under proposed/GDPR-style measures rising to −1.1% under economy-wide localization; Bauer, Lee-Makiyama, van der Marel & Verschelde 2014, ECIPE Occasional Paper No. 3/2014) yet quantify no realized sovereignty benefit, and chip export controls — the other main instrument — show contested efficacy: one cross-firm study finds no innovation harm to 30 leading semiconductor firms (Schumacher 2024, CSIS) while case evidence documents systematic circumvention via software/efficiency gains and chip exfiltration/smuggling (Gupta, Walker & Reddie 2024). No replicated study demonstrates that any sovereign-AI regime measurably delivers the jurisdictional control it asserts.","sources":["Bauer, Lee-Makiyama, van der Marel & Verschelde 2014 (ECIPE Occasional Paper No. 3/2014, 'The Costs of Data Localisation: Friendly Fire on Economic Recovery')","Schumacher 2024 (CSIS, 'Did U.S. Semiconductor Export Controls Harm Innovation?')","Gupta, Walker & Reddie 2024 (arXiv:2411.14425, 'Whack-a-Chip: The Futility of Hardware-Centric Export Controls')"]},"wikiUrl":"https://policywindow.org/wiki/sovereign-ai#social-science-evidence"},{"code":"synthetic_content_provenance","label":"Synthetic Content Provenance","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"The harm provenance targets is real but concentrated, and the technical premise that the mandated signal survives is itself empirically shaky. Synthetic-media harm is well documented in two domains: non-consensual intimate imagery (Ajder et al.'s 2019 Deeptrace audit found 96% of deepfake videos were pornographic and effectively 100% targeted women) and impersonation fraud (the Arup case, ~US$25.6M / HK$200M lost via a deepfake video call). The honest caveat is twofold: a feared broad political-misinformation harm is not yet demonstrated at scale, and CS work shows invisible watermarks are removable in practice (Jiang, Zhang & Gong 2023, WEvade, evade detection via adversarial perturbation; Zhao et al. 2024 prove pixel-level watermarks are provably removable via regeneration attacks), so the provenance signal a rule would mandate is itself contested.","sources":["Ajder, Patrini, Cavalli & Cullen 2019 (Deeptrace, 'The State of Deepfakes: Landscape, Threats, and Impact')","Jiang, Zhang & Gong 2023 ('Evading Watermark based Detection of AI-Generated Content', ACM CCS 2023)","Zhao et al. 2024 (NeurIPS, 'Invisible Image Watermarks Are Provably Removable Using Generative AI')","Arup deepfake fraud (CNN Business, 2024-05-16, US$25.6M)"]},"governanceEfficacy":{"epistemicStatus":"thin","finding":"There is no impact evaluation showing that mandated provenance/labeling reduces synthetic-media harm; the major mandates (China's GenAI labeling Measures, effective 2025-09-01; EU AIA Art. 50, machine-readable marking) are too new and unevaluated, and the delivery layer is leaky: the C2PA spec's own Security Considerations document the strip-and-repost threat, and platform audits report C2PA/Content-Credentials metadata is stripped by essentially all major social platforms on upload (consistent with Imatag's 2018 finding that ~80% of uploaded images lose metadata, only ~15% retaining it). The closest analogue evaluation literature — Pennycook, Bear, Collins & Rand (2020), the 'implied truth effect' — gives reason for caution rather than confidence: labeling only some content can make unlabeled false content seem more credible, so a partial-coverage provenance regime could backfire.","sources":["Pennycook, Bear, Collins & Rand 2020 (Management Science 66(11):4944-4957, 'The Implied Truth Effect')","China Measures for Labeling AI-Generated Synthetic Content (eff. 2025-09-01); EU AI Act Art. 50","Imatag 2018 metadata-stripping study (~80%); C2PA Security Considerations (spec.c2pa.org) on manifest removal"]},"wikiUrl":"https://policywindow.org/wiki/synthetic-content-provenance#social-science-evidence"},{"code":"tech_sovereignty","label":"Technological Sovereignty","kind":"political_frame","empiricalConsensus":"emerging","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"The structural fact that compute capacity is geographically concentrated is well-measured: Lehdonvirta, Wú & Hawkins find only ~33 countries host facilities with AI-accelerator hardware and roughly 24 have the capacity to train full-scale foundation models, the Stanford AI Index 2026 reports low-income countries collectively hold ~0.1% of global data-centre compute (the US hosting >10x any other nation), and Cottier et al. document amortized frontier-training cost rising 2.4x/year (95% CI 2.0-3.1x) toward $1B+ models by 2027. But this is a political-economy FRAME, not a documented harm, and the core contested claim of the topic, that the cost curve locks mid-sized economies OUT of capability, is empirically cut both ways: a feasibility study of Brazil and Mexico (Malagon et al. 2025) estimates usable (non-frontier) 10-trillion-token sovereign models are fiscally viable at roughly $8-14M on H100 hardware, and DeepSeek-style efficiency gains (V3 trained for ~$5.5M, ~11x less compute than Llama 3 405B) show frontier-adjacent performance at a fraction of prior compute, so whether domestic frontier-tier capability is foreclosed for middle powers remains genuinely unsettled.","sources":["Lehdonvirta, Wú & Hawkins 2024 (Compute North vs. Compute South, Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics & Society 7:828-838)","Cottier, Rahman, Fattorini, Maslej & Owen 2024 (The Rising Costs of Training Frontier AI Models, arXiv:2405.21015)","Stanford AI Index 2026 (Maslej et al., Stanford HAI); Malagon, Ulloa Ruiz, Sandoval Plaza, Rosario Bolívar, García Mesa & Alvarado Morales 2025 (The Feasibility of Training Sovereign Language Models in the Global South: A Study of Brazil and Mexico, arXiv:2510.19801)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that technological-sovereignty policies (on-shore compute mandates, national foundation-model champions, talent-retention schemes such as EuroHPC AI Factories or India's IndiaAI Mission) actually deliver sustained domestic capability or strategic autonomy; these programs are recent, utilization and cost-per-GPU-hour are largely unpublished, and no counterfactual study exists. The closest analogue evidence base, the industrial-policy literature synthesized by Juhász, Lane & Rodrik, finds that properly-identified studies are more favorable than older correlational work suggested but that outcomes depend heavily on instrument design and structural context, and the older national-champion record warns of subsidized 'zombie' firms and government capture, so the closest analogue is mixed and the direct evidence that the sovereignty rule works is simply missing.","sources":["Juhász, Lane & Rodrik 2024 (The New Economics of Industrial Policy, Annual Review of Economics 16:213-242)","Ahmed & Wahed 2020 (The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research, arXiv:2010.15581)","IndiaAI Mission (Indian Cabinet, March 2024); EuroHPC Joint Undertaking AI Factories (2024 regulation amendment; no published impact evaluation)"]},"wikiUrl":"https://policywindow.org/wiki/tech-sovereignty#social-science-evidence"},{"code":"training_data","label":"Training-Data Rights","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"That foundation models ingest copyrighted and personal works without consent is undisputed; whether that ingestion produces legally cognizable reproduction harm is genuinely contested. The CS evidence that models can memorize and emit verbatim training text is robust and replicated — Carlini et al. (2021) extracted hundreds of verbatim sequences (including PII) from GPT-2, and follow-up work (Carlini et al., Quantifying Memorization, ICLR 2023) showed extraction scales log-linearly with model size and with example duplication. Honest caveat: verbatim reproduction is the exception, not the norm — the UK High Court held that Stable Diffusion's model weights never stored copies of the training images (defeating the secondary-infringement theory), and Getty abandoned its primary training-infringement claim at trial for lack of evidence, so whether the empirical phenomenon amounts to actionable harm (rather than transient, non-expressive use) remains the open question driving NYT v. OpenAI and parallel regimes.","sources":["Carlini, Tramèr, Wallace, Jagielski, Herbert-Voss, Lee, Roberts, Brown, Song, Erlingsson, Oprea & Raffel 2021 (Extracting Training Data from Large Language Models, 30th USENIX Security Symposium)","Carlini, Ippolito, Jagielski, Lee, Tramèr & Zhang 2023 (Quantifying Memorization Across Neural Language Models, ICLR 2023; arXiv:2202.07646)","Getty Images (US) Inc & ors v Stability AI Ltd [2025] EWHC 2863 (Ch) (UK High Court, 4 Nov 2025 — no secondary infringement; primary training claim abandoned at trial)","The New York Times Co. v. Microsoft Corp. & OpenAI (S.D.N.Y., No. 1:23-cv-11195; consolidated In re OpenAI Copyright Infringement Litigation, Apr. 2025; ongoing 2025-2026)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no impact evaluation showing that the CDSM Directive Article 4 TDM exception plus its Article 4(3) opt-out reservation regime actually reduces unlicensed ingestion or channels compensation to rightsholders — the evidence that the rule works as designed is itself missing. The only available evidence is early case law and doctrinal scholarship, which document the mechanism's contested operation rather than its success: in Kneschke v. LAION the Hamburg Higher Regional Court (on appeal, 10 Dec 2025) held that a rights reservation in natural language did NOT satisfy Article 4(3)'s machine-readability requirement, invalidating the opt-out (note: the first-instance Regional Court had left the Article 4 question largely open and the case ultimately turned on the Article 3 scientific-research exception, so this machine-readability holding is appellate and not yet settled — a further appeal to the Federal Court of Justice was permitted). Legal scholars characterize the Article 4 opt-out as practically difficult and unharmonized, with no observed market in TDM licences or systematic enforcement to evaluate.","sources":["Kneschke v. LAION (Hamburg Regional Court, 27 Sept 2024, 310 O 227/23; on appeal Hamburg Higher Regional Court, 10 Dec 2025, 5 U 104/24 — opt-out held not machine-readable; further appeal to BGH permitted)","Margoni & Kretschmer 2022 (A Deeper Look into the EU Text and Data Mining Exceptions, GRUR International 71(8):685-701)","Quintais 2025 (Generative AI, Copyright and the AI Act, Computer Law & Security Review 56:106107)"]},"wikiUrl":"https://policywindow.org/wiki/training-data#social-science-evidence"},{"code":"transparency","label":"Transparency Obligations","kind":"procedural","empiricalConsensus":"contested","realHarmUnprovenGovernance":true,"problemReality":{"epistemicStatus":"contested","finding":"Documentation artifacts (model cards, datasheets) are well-specified as proposals and are genuinely adopted, but the empirical premise that mandated disclosure produces meaningful transparency is contested. Selbst & Barocas (2018) argue inscrutability and non-intuitiveness are distinct problems and that disclosing rules does not resolve the latter, and large-scale audits find documentation is sparsely and unevenly completed: a systematic analysis of 32,111 Hugging Face model cards (Liang et al. 2024) found environmental-impact, limitations and evaluation sections least often filled, and Bhat et al. (2023, 45 practitioners) found a substantial gap between the documentation proposal and actual practice. Honest caveat: the documentation frameworks themselves are real and adopted, so the dispute is about whether disclosure conveys decision-relevant information, not whether the artifacts exist.","sources":["Selbst & Barocas 2018 (Fordham Law Review 87:1085-1139)","Liang et al. 2024 (Nature Machine Intelligence, s42256-024-00857-z, 'Systematic analysis of 32,111 AI model cards')","Bhat et al. 2023 (CHI '23, 'Aspirations and Practice of ML Model Documentation', DOI 10.1145/3544548.3581518)","Mitchell et al. 2019 (FAccT, Model Cards for Model Reporting); Gebru et al. 2021 (CACM 64(12):86-92, Datasheets for Datasets)"]},"governanceEfficacy":{"epistemicStatus":"absent","finding":"There is no rigorous impact evaluation showing that AI transparency mandates (model cards, training-data summaries) measurably reduce bias, misuse or accidents — the central regulatory assumption is empirically untested, partly because flagship mandates like EU AI Act Art. 53(1)(d) GPAI training-data summaries are only subject to AI Office enforcement/verification from 2 August 2026 (the obligation itself began 2 August 2025 for new models). The closest analogue, mandated consumer disclosure, shows small and context-dependent effects: Bollinger, Leslie & Sorensen (2011) found mandatory calorie posting cut average calories per transaction by about 6%, while Loewenstein, Sunstein & Golman (2014) review evidence that disclosure effects are frequently diminished or even reversed by limited attention and often change provider rather than recipient behavior. These are analogues, not AI studies; no study demonstrates that AI transparency disclosure achieves its stated downstream safety aims.","sources":["Bollinger, Leslie & Sorensen 2011 (AEJ: Economic Policy 3(1):91-128)","Loewenstein, Sunstein & Golman 2014 (Annual Review of Economics 6:391-419, 'Disclosure: Psychology Changes Everything')","EU AI Act Art. 53(1)(d) GPAI training-data summary (obligation from 2 Aug 2025; AI Office enforcement from 2 Aug 2026)"]},"wikiUrl":"https://policywindow.org/wiki/transparency#social-science-evidence"}],"license":"https://creativecommons.org/licenses/by/4.0/","attribution":"Source: Policy Window AI Governance Catalog (https://policywindow.org/wiki). Findings are AI-drafted under charter §7.13, epistemic-honesty reviewed, and citation-corroborated; content CC BY 4.0.","note":"Deterministic aggregate over the published catalog — re-derivable from the per-topic evidenceBase. Every finding is sourced; a thin/absent finding is itself a published result (the second silence)."}