A new analysis published by OpenAI has identified significant reliability issues in SWE-Bench Pro, a benchmark used to evaluate the coding capabilities of large language models on real-world GitHub issues.
According to OpenAI's writeup, the team manually inspected a sample of benchmark failures and found that a large share were not due to flaws in model-generated code, but to problems with the benchmark itself — what the researchers call "noise."
Sources of Noise
OpenAI's analysis categorizes the causes of these false failures:
- Flaky tests: Some tests in the evaluation environment pass or fail intermittently, regardless of the submitted code.
- Environment mismatches: Differences between the benchmark's execution environment and the original GitHub repository caused otherwise correct code to fail.
- Ambiguous problem statements: Incomplete or unclear issue descriptions led models to produce valid solutions that were marked incorrect.
- Incorrect ground truth: In some cases, the benchmark's reference solution was itself flawed.
OpenAI reports that roughly 40% of failures initially attributed to model shortcomings were actually caused by these benchmark defects rather than genuine coding errors.
Why It Matters
Benchmarks like SWE-Bench Pro are widely used to track and compare coding performance across models. If a large fraction of "failures" actually reflect flaws in test design rather than model ability, published comparisons and leaderboards built on such benchmarks may be misleading. OpenAI's post argues for more rigorous, human-reviewed validation when constructing coding benchmarks, to separate genuine model errors from artifacts of a noisy evaluation environment.