This article is part of our AI Agent Guides. See how models compare this week on the AI Agent Leaderboard, a ranking of rankings refreshed every week.
SWE-bench is a benchmark that asks AI to fix real software issues taken from GitHub, then runs tests to see whether the fix works. Be careful with its best-known version, SWE-bench Verified. OpenAI helped create it, and in 2026 it said Verified no longer measures frontier coding ability.
If a vendor quotes a Verified score today, treat it as old background. For newer coding results, look at other boards, including SWE-Bench Pro.
Key Takeaways
- SWE-bench gives a model a real codebase and issue. It passes only if the target tests pass and existing ones still do.
- SWE-bench Verified is a 500-task, human-filtered subset, released in August 2024.
- In February 2026, OpenAI said Verified was contaminated and that many audited tasks had flawed tests.
- SWE-Bench Pro, from Scale, was built to resist contamination, with 1,865 tasks. OpenAI says it is better, though not perfect.
- For current coding results, see our coding board, which does not include SWE-bench in the average.
What SWE-bench tests
"SWE" stands for software engineering. The original 2023 paper built 2,294 problems from real GitHub issues and the pull requests that fixed them, across 12 popular Python repositories. A pull request is a proposed code change.
The model receives the codebase and the issue text, and it never sees the tests. OpenAI's summary explains the check: new tests must go from failing to passing, and regression tests must keep passing. All tests must pass for credit.
It was a good idea, using real problems with objective grading. For a time it became a standard number in model launches. Then AI improved, and the flaws started to dominate.
The SWE-bench family
The name now covers several datasets. These counts come from the SWE-bench site and Scale's Pro page, checked on 2 November 2026.
| Version | Tasks | Run by | What is different |
|---|---|---|---|
| SWE-bench (original) | 2,294 | SWE-bench team | 12 Python repositories |
| Lite | 300 | SWE-bench team | A cheaper subset |
| Verified | 500 | SWE-bench team with OpenAI | Filtered by human reviewers |
| Multilingual | 300 | SWE-bench team | 42 repositories across 9 languages |
| Multimodal | 480 | SWE-bench team | Issues described with visual elements |
| SWE-Bench Pro | 1,865 | Scale AI | Harder tasks, copyleft and private code |
The SWE-bench site also lists a "Bash Only" view of Verified, in which every model runs in the same mini-SWE-agent environment. Harnesses matter on these boards, and our guide to harnesses explains why.
What contamination means
Contamination happens when a model has seen a test's questions or answers during training. It is like giving students the exam the night before. They may not memorize it, but they will do better than students who never saw it, and the score stops measuring what you wanted.
SWE-bench was especially exposed for an obvious reason. Its problems come from popular open-source projects, and OpenAI notes that many model providers train on those same repositories. Nobody had to cheat on purpose for contamination to occur.
Vals AI lists leakage as a major problem across AI evaluation, not just here. That is why it keeps its proprietary test sets private. See our Vals Index guide for how that works.
What OpenAI found
OpenAI published its reasoning on 23 February 2026. It reported two problems, and both matter for anyone reading coding scores.
Flawed tests. OpenAI audited 138 Verified problems that one of its models did not solve consistently. It found material issues in 59.4% of them. That included 35.5% of audited tasks with overly narrow tests and 18.8% with tests checking for things the problem never asked for.
So a model could write a correct fix and still fail. One example needed a function with a specific name that the problem statement never mentioned.
Training exposure. OpenAI used automated red-teaming to probe three frontier models for what they knew about the tasks. It reports that all three could reproduce the original fix or exact problem details for some tasks. That indicates they saw them in training.
OpenAI's conclusion was blunt: improvements on Verified increasingly reflect exposure to the benchmark, not real software ability. It stopped reporting those scores and advised other developers to do the same.
Keep the source in mind, since OpenAI competes with the labs it tested. Vals also removed SWE-bench Verified from its index in August 2026, though it gave saturation as the reason. Saturated means the test no longer separates models well.
How SWE-Bench Pro tries to fix this
Scale's SWE-Bench Pro targets those problems in its design. Its paper says the tasks may take a professional engineer hours to days. Scale reports that reference fixes average 107.4 lines across 4.1 files.
| Feature | SWE-bench Verified | SWE-Bench Pro |
|---|---|---|
| Tasks | 500 | 1,865, across 41 repositories |
| Source code | Popular open-source Python projects | GPL-style copyleft repositories, plus private startup code |
| Sets | One public set | Public (731) and private (276) leaderboards, with 858 held out |
| Human work | Filtering out bad tasks | Rewriting unclear tasks so they can be solved, and checking tests |
The private set is the interesting part. It comes from 18 proprietary codebases that outsiders cannot see, so training exposure is much less likely. Scale reports that scores drop on that private set, which it reads as a more realistic test of generalization.
Copyleft licenses reduce the risk without removing it. OpenAI's check found some contamination on Pro, though rarer and less egregious, and no model could produce a complete verbatim fix. OpenAI calls Pro "not perfect".
Reading the boards today
Here are the checks we would use before relying on any of these numbers.
- Which dataset? Verified, Lite and Pro are not interchangeable. One model can post very different numbers on each.
- Which scaffold? Scale marks some rows as run on a different harness, with different cost and turn limits on others.
- Who ran it? The SWE-bench board marks runs performed or checked by its team. Self-reported results deserve less weight.
- How old is the text? Scale's explanatory paragraphs discuss older models, while its table lists newer ones. Go by the table and its date.
- What is the error range? Scale shows a margin on each row and a rank based on upper confidence bounds.
Why it is not in our average
Our methodology page requires a board that tests agent work, publishes openly and has been updated in the past 120 days. On the coding board, we leave out SWE-bench because many Verified scores are older or self-reported.
That is not a verdict that SWE-Bench Pro is poor. We give our reasoning so that you can disagree. See the main board for the current inputs.
Frequently asked questions
Is SWE-bench still useful?
Partly. The family is still useful for research and for comparing cost or setup, and Arena's harness study used SWE-bench Lite in 2026. Be cautious about using old Verified scores to pick a frontier model.
What is the difference between SWE-bench Verified and Pro?
Verified is a human-filtered 500-task subset drawn from popular Python projects. Pro is a separate, larger benchmark from Scale. It uses copyleft and private code with human-augmented tasks and aims to reduce contamination.
Did AI labs cheat on SWE-bench?
OpenAI's note does not claim that. It says public repositories end up in training data, which is hard to avoid. For deliberate cheating cases on other boards, see can AI benchmarks be gamed.
Should my team use a coding benchmark to pick a tool?
Use it to shortlist. Then test on your own repository with your own tests, since Scale reports that difficulty varies heavily by repository. Our list of coding agents with MCP is a good next read.
