SWE-bench Verified
SWE-bench: Resolving Real-World GitHub Issues (Verified Subset)
Standard benchmark evaluating an AI model's ability to solve end-to-end software engineering problems from real GitHub issues in popular Python repositories.
Verified Model Leaderboard
Ranked by verified Pass@1 (%) score from official technical reports.
| Rank | AI Model & Provider | Score (Pass@1 (%)) | Evaluation Methodology | Verification Date | Source |
|---|---|---|---|---|---|
| #1 | Claude 3.7 Sonnet Anthropic • Hybrid Reasoning Transformer | 70.3% | Pass@1, Thinking mode enabled with Claude Code scaffolding and 64k token thinking budget | Feb 2025 | Anthropic Claude 3.7 Sonnet Announcement |
| #2 | OpenAI o3-mini OpenAI • Reinforcement Learning Reasoning Model | 49.3% | Pass@1 with high reasoning effort | Jan 2025 | OpenAI o3-mini Announcement |
| #3 | DeepSeek-R1 DeepSeek • MoE (Mixture of Experts) Reasoning | 49.2% | Pass@1 with standard SWE-bench framework | Jan 2025 | DeepSeek-R1 Technical Report |
| #4 | Claude 3.5 Sonnet (v2) Anthropic • Dense Transformer | 49.0% | Pass@1, standard agentic scaffolding without reasoning tokens | Oct 2024 | Anthropic Technical Announcement |
| #5 | OpenAI o1 OpenAI • Reinforcement Learning Reasoning Model | 48.9% | Pass@1, high reasoning effort without external agent scaffolding | Dec 2024 | OpenAI o1 System Card |
| #6 | Claude 3.5 Haiku Anthropic • Dense Transformer | 40.6% | Pass@1 with standard developer scaffold | Nov 2024 | Anthropic Announcement |
| #7 | GPT-4o OpenAI • Multimodal Transformer | 38.8% | Pass@1 standard harness | Aug 2024 | SWE-bench Official Leaderboard |
| #8 | Qwen 2.5 Coder 32B Instruct Qwen (Alibaba Cloud) • Dense Transformer | 35.8% | Pass@1 with standard open-weights harness | Nov 2024 | Qwen 2.5 Coder Technical Report |
| #9 | GPT-4o mini OpenAI • Multimodal Small Transformer | 20.2% | Pass@1 standard scaffold | Jul 2024 | SWE-bench Leaderboard |
Evaluation Methodology & Harness
Verified subset filters out 500 hand-checked, unambiguous problem instances with comprehensive unit tests executed in sandboxed Docker containers.
500 verified real-world GitHub issues across popular open-source repositories including Django, SymPy, scikit-learn, and pytest.
Why This Benchmark Matters
The gold standard for evaluating real-world coding agents, requiring multi-file codebase navigation, bug localization, and patch generation.
Known Limitations & Contamination Risks
Focused primarily on Python; execution speed and test-harness timeouts can affect reproducibility without standardized scaffolding.
Data Accuracy & Verification Notice
AI model specifications, pricing records, and benchmark metrics published on this platform are compiled directly from authoritative sources (official provider documentation, research papers, and verified evaluation harnesses). Benchmark results reflect specific test harnesses and prompting methodologies; scores are not directly comparable across differing evaluation setups.