OpenAI says SWE-Bench Pro has enough broken tasks that model developers should treat its results carefully and stop relying on OpenAI’s earlier recommendation to adopt it.
The July 8 audit reviewed SWE-Bench Pro, a coding-agent benchmark designed to improve on SWE-bench Verified with longer-horizon and more realistic software tasks. OpenAI says frontier models improved from 23.3% to 80.3% pass rate on the benchmark’s 731-task public split in eight months, which made the benchmark look like a strong signal for coding-agent progress.
The audit changes that read. OpenAI’s datapoint analysis pipeline flagged 200 broken tasks, or 27.4% of the public split. A separate human annotation campaign identified 249 broken tasks, or 34.1%. OpenAI’s current estimate is that roughly 30% of SWE-Bench Pro tasks are broken.
The failures are benchmark design failures, not just model misses
OpenAI groups the problems into four main categories: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.
Those failures matter because repository-level coding benchmarks often hide the tests used for grading. If a prompt leaves out a requirement enforced by hidden tests, a model can produce a reasonable fix and still fail. If a test is too narrow, a model can pass while leaving the real feature incomplete.
That turns a benchmark leaderboard into a mixed measurement. Some scores reflect model capability. Some reflect whether the benchmark’s prompt, tests, and reference fix form a fair task.
OpenAI also says the audit used agents to inspect task metadata, model attempts, and failure traces before human review. That is a notable second-order point: stronger coding agents are now being used to audit the evaluations that measure coding agents.
Coding-agent buyers should ask what the tests prove
The practical lesson is not that all coding-agent benchmarks are useless. It is that benchmark names are no longer enough.
Teams comparing coding agents should ask how tasks were sourced, whether prompts were written for evaluation or copied from messy development history, how hidden tests were checked, and whether failures were audited by humans with repository context.
For internal pilots, the safer pattern is smaller but inspectable task sets: real issues from the team’s own repositories, explicit acceptance criteria, visible review rubrics, and hidden regression tests only after the prompt is checked for fairness.
