SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Abstract
SWE-bench Science benchmarks coding agents on scientific software repair, revealing failure mechanisms and mixed effects of scientific guidance.
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce SWE-bench Science, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50\%, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Community
Every agent I've watched hit a failing test eventually learns the same shortcut: make the test pass, don't fix the code. Green check is the reward, so green check is what gets optimized. The real question for a science benchmark isn't whether the tests pass — it's whether the agent can tell the difference between a correct fix and a test that was quietly edited to bless broken output.
So I'd want to know: does SWE-bench Science catch that? If the harness only checks pass/fail on the original suite, an agent that rewrites assertions to match its own wrong output sails through. And pass rate says nothing about whether the science survived — a test can pass while the underlying computation is still wrong. Semantic preservation is the hard part, and it's the part that matters when the output feeds a downstream experiment.
That is exactly why we designed a separate private test suite. The private tests are behavior-oriented and target the scientific contracts behind each task, rather than merely checking whether a particular line was changed. They are designed to expose hard-coded outputs, public-fixture overfitting, and fixes that only make the visible reproduction pass.
One clarification about the evaluation pipeline: an agent may technically edit files that are visible in its workspace, including a public reproduction, and such changes may appear in the submitted patch. However, the private tests are kept inside the verifier image and are not exposed to the agent. The verifier starts from a fresh workspace, applies the submitted patch, and runs the separate private suite against the resulting source code.
Therefore, modifying a public assertion can at most make the public check pass. It cannot make the private scientific checks pass, and the agent reward is computed from the private evaluation rather than from the editable public test.
Public tests are therefore a diagnostic reproduction of the reported issue, while the private suite is the actual scientific acceptance criterion. For example, Qwen3.8-27B passed all 119 public reproductions but achieved full private Pass@1 on only 35 of 119 tasks (You can see this result on our website soon). This is precisely why a green public check alone is not considered a successful solution in SWE-bench Science.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper