OpenAI Finds Broken Tasks in SWE-Bench Pro
OpenAI said on July 8, 2026 that roughly 30% of SWE-Bench Pro tasks are broken after an audit of the coding-agent benchmark. The company reports that pass rates on the 731-task public split rose from 23.3% to 80.3% in eight months, then found task-quality defects that can distort both failures and successes. For teams using benchmark scores as buying, routing, or release gates, the practical lesson is that dataset quality now matters as much as model quality. OpenAI says the issues include overly strict hidden tests, underspecified prompts, low-coverage tests, and one misleading prompt, and it has retracted its earlier recommendation to adopt SWE-Bench Pro.
Coding-agent benchmarks are becoming procurement signals, launch claims, and internal release gates, so a benchmark-quality failure can mislead both buyers and model teams. The useful lesson from OpenAI's audit is not that SWE-style evaluations should be ignored; it is that realistic software tasks need the same data-quality discipline as any production evaluation set.
What happened
OpenAI published a July 8, 2026 audit of SWE-Bench Pro, the coding-agent benchmark designed for longer-horizon software-engineering tasks. The company says frontier-model pass rates on the 731-task public split rose from 23.3% to 80.3% in eight months, then used a datapoint-analysis pipeline and human review to inspect whether the tasks still measured real capability. OpenAI says its pipeline flagged 200 tasks, or 27.4%, as broken, while a parallel human annotation campaign marked 249 tasks, or 34.1%, as broken. Its overall estimate is that roughly 30% of the benchmark has breaking issues.
Technical context
The reported failure modes are familiar to engineers who maintain test suites. Overly strict tests can reject functionally correct solutions because they enforce unstated implementation details. Underspecified prompts can make hidden requirements impossible to infer. Low-coverage tests can let incomplete patches pass. A misleading prompt can point a model toward behavior that conflicts with the evaluator. Each defect changes what a pass or fail means.
For practitioners
Teams evaluating coding agents should treat public benchmark movement as a triage signal, not proof of production readiness. Before turning benchmark scores into vendor decisions or release gates, audit task construction, hidden-test coverage, contamination risk, and whether the tasks resemble the organization's actual repositories and review process.
What to watch
OpenAI says it is retracting its earlier recommendation to adopt SWE-Bench Pro and wants benchmarks built with stronger human oversight. Watch whether model labs, eval vendors, and enterprise buyers start publishing task-quality audits alongside headline coding-agent scores.
Key Points
- 1OpenAI estimates about 30% of SWE-Bench Pro tasks are broken after combining agent-assisted and human-review audits.
- 2The cited defects can make coding-agent scores reflect hidden-test artifacts instead of genuine software-engineering capability.
- 3Evaluation teams should audit task quality and validate benchmark gains against internal repositories before using scores as gates.
Scoring Rationale
The finding is important because SWE-Bench Pro affects how labs, vendors, and buyers interpret coding-agent progress, and OpenAI is retracting a prior recommendation after finding material task defects. It remains below industry-shaking because the change concerns evaluation validity rather than a new model capability, platform launch, or regulation.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
