Converging AI and Human Roles

Study · Tech & Engineering

SWE-bench Verified

SWE-bench is a benchmark that asks a language model to resolve real software issues. Its creators introduced it in a paper submitted on October 10, 2023 (presented at ICLR 2024) with 2,294 software engineering problems drawn from real GitHub issues and pull requests across 12 popular Python repositories; given a codebase and an issue description, the model must edit the code to fix it. The best model in that paper solved 1.96% of the issues.1

SWE-bench Verified is a human-validated subset of 500 of those samples, annotated by 93 software developers, with each sample reviewed by three annotators for a clear issue description and appropriate unit tests.2 Epoch AI runs it independently on 484 of the 500 samples that validate on its infrastructure; in its data the highest score is 83.5%, set by Claude Opus 4.7, and our ranking lists the top 25.23 The benchmark’s own maintainers publish a separate leaderboard at swebench.com.4

Its limits are documented. Epoch had estimated an error rate of 5 to 10% in the samples, and its September 3, 2026 review rated the benchmark “Flawed”, citing an OpenAI audit in which 59.4% of the audited tasks had test cases that rejected functionally correct submissions, and the likelihood that models have seen the codebases in training.25 It measures patches to isolated issues, not developer productivity: for trial evidence on that, see Do coding agents make developers faster?

Illustration: SWE-bench Verified, a benchmark of real GitHub issues that models must resolve by editing code.

Key facts

Original benchmarkSWE-bench, 2,294 problems from 12 Python repositories (arXiv October 10, 2023; ICLR 2024)1
Verified subset500 human-validated samples, reviewed by 93 developers2
Epoch AI evaluation484 of 500 samples; highest score 83.5% (Claude Opus 4.7) in its data of October 20, 202623
Epoch error-rate estimate5 to 10% of samples2
Epoch verdict“Flawed” (September 3, 2026)5

Appears in

Sources

  1. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv:2310.06770, October 10, 2023 (last revised November 11, 2024).
  2. Epoch AI, SWE-bench Verified (benchmark page and methodology), retrieved 2026-10-20.
  3. Epoch AI, Capabilities & benchmarking data (swe_bench_verified.csv, CC BY 4.0), retrieved 2026-10-20.
  4. SWE-bench maintainers, SWE-bench leaderboards, retrieved 2026-10-20.
  5. Epoch AI, SWE-bench Verified: Benchmark review, September 3, 2026.