?asOf= parameter to see the current catalog state.What it measures
Solve real-world GitHub issues from 12 popular Python repos. The 'Verified' subset is human-validated to remove ambiguity and have working tests.
500-task verified subset. Run-time evaluation; can't be gamed by pure memorisation but agent harness affects results.
Results & interpretation
Claimed scores
| Model | Score | Claim type | Reported | Citation |
|---|---|---|---|---|
| claude-opus-4-7 | 78.4 % solved | vendor card | 2025-05-22 | Anthropic model card |
How to read this number
Contamination risk: medium
Some test items may leak into training corpora; treat headline scores with mild skepticism and prefer evaluation runs with held-out subsets.
What a high score does and does not establish. A score evidences performance on this benchmark’s specific construct under its specific format; it is not, on its own, evidence of general capability, reliable real-world task performance, or safety.
The second silence. evidence: thin The evidence that a benchmark score predicts real-world deployment outcomes (construct-to-deployment validity) is sparse; benchmark performance and deployed performance are not established to be the same thing, and contamination can inflate the headline figure above true held-out ability.
Governance relevance
A benchmark measures a capability; governance attaches to the topicsthat capability bears on. These topic articles carry the instrument×dimension coverage matrix and the social-science so-what for this domain.
- Foundation Models / GPAI— coverage matrix + does-governance-work evidence
- Agentic AI Governance— coverage matrix + does-governance-work evidence
Further reading
50 academic & grey-literature sources on the governance questions this benchmark's results inform — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Governing AI Agents Preprint✦ AIUses "agency law and theory to identify and characterize problems arising from AI agents" and proposes governance infrastructure built on inclusivity, visibility, and liability.
- Infrastructure for AI Agents Peer-reviewed✦ AIProposes "agent infrastructure": external technical systems for attributing actions "to specific agents, their users, or other actors," shaping interactions, and remediating harms.
- Multi-Agent Risks from Advanced AI Research institute✦ AIIdentifies three failure modes of advanced multi-agent systems — "miscoordination, conflict, and collusion" — plus seven risk factors, posing challenges distinct from single-agent AI.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Authenticated Delegation and Authorized AI Agents Preprint✦ AIIntroduces a framework for authenticated, authorized, and auditable delegation to AI agents by extending OAuth 2.0/OpenID Connect, maintaining accountability chains for agent actions.
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents Peer-reviewed✦ AIProvides a 440-task benchmark across 11 harm categories measuring whether LLM agents resist or comply with harmful multi-step tool-use tasks, grounding safety-evaluation regimes for agents.
- Better together? Human oversight as means to achieve fairness in the European AI Act governance Peer-reviewed✦ AIExamines whether Article-14 human oversight of high-risk/autonomous AI can actually deliver fairness, probing the limits of human-in-the-loop as a governance mechanism.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Visibility into AI Agents Peer-reviewed✦ AIProposes agent identifiers, real-time monitoring and activity logs to give governance actors visibility — "where, why, how, and by whom certain AI agents are used."
+ 38 more on these governance questions — see the literature index.
References
The primary instrument sources behind the article's classifications.
How to cite this benchmark
Use the primary methodology source for academic citations; reference the Policy Window article for the cross-model leaderboard.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/swe-bench-verified — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here