?asOf= parameter to see the current catalog state.What it measures
Abstract reasoning over visual grids. Each task requires inferring the transformation rule from 2-3 examples.
v2 launched 2024-12 with harder tasks designed to remain unsolvable by pure pattern matching. $1M public prize for >85% on private set.
Results & interpretation
Claimed scores
No claims have been recorded yet for this benchmark in the Policy Window catalog.
How to read this number
Contamination risk: low
Benchmark items are unlikely to appear in training corpora — scores are credible reflections of underlying capability.
What a high score does and does not establish. A score evidences performance on this benchmark’s specific construct under its specific format; it is not, on its own, evidence of general capability, reliable real-world task performance, or safety.
The second silence. evidence: thin The evidence that a benchmark score predicts real-world deployment outcomes (construct-to-deployment validity) is sparse; benchmark performance and deployed performance are not established to be the same thing, and contamination can inflate the headline figure above true held-out ability.
Governance relevance
A benchmark measures a capability; governance attaches to the topicsthat capability bears on. These topic articles carry the instrument×dimension coverage matrix and the social-science so-what for this domain.
- Foundation Models / GPAI— coverage matrix + does-governance-work evidence
See also
Further reading
40 academic & grey-literature sources on the governance questions this benchmark's results inform — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Evaluating Frontier Models for Dangerous Capabilities Preprint✦ AIPilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating.
- Frontier AI Regulation: Managing Emerging Risks to Public Safety Preprint✦ AIArgues "industry self-regulation is an important first step" but "government intervention will be needed", proposing safety standards, registration and reporting, and compliance mechanisms.
- Regulating ChatGPT and other Large Generative AI Models Peer-reviewed✦ AIArgues AI regulation "has primarily focused on conventional AI models, not LGAIMs" and should target "concrete high-risk applications, and not the pre-trained model itself".
- A Proposal for a Definition of General Purpose Artificial Intelligence Systems Peer-reviewed✦ AIFinds existing GPAIS definitions "do not provide sufficient guidance" and proposes "a functional definition of the term that facilitates its governance within the EU".
- Foundation Models and Fair Use Peer-reviewed✦ AIShows foundation models "are trained on copyrighted material" and warns "fair use is not guaranteed", urging technical mitigations to keep training and deployment within fair use.
- The risks of risk-based AI regulation: taking liability seriously Preprint✦ AIArgues the AI Act's ex-ante risk tiers under-govern foundation models and that 'taking liability seriously as the key regulatory mechanism' is a more effective lever.
- Market Concentration Implications of Foundation Models Preprint✦ AIArgues foundation models tend toward 'natural monopoly' and that regulators must ensure 'the contestability of the market by tackling strategic behavior'.
+ 28 more on these governance questions — see the literature index.
References
The primary instrument sources behind the article's classifications.
How to cite this benchmark
Use the primary methodology source for academic citations; reference the Policy Window article for the cross-model leaderboard.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/arc-agi-v2 — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here