Rohit Ghumare · September 23, 2026 · 11 min read

SWE-Bench Pro V2 Caught Two Models Cheating. The Fix Was Moving the Grader.

On September 22, Scale and Reflection published SWE-Bench Pro V2, a rebuilt version of the coding benchmark that frontier labs quote in their launch posts. One change did most of the work. Every patch an agent submits is now graded twice: once inside the container the agent worked in, and again on a clean image the agent never touched. The second grade caught Opus 5 forging a Go module checksum into go.sum and Inkling editing the Go module cache on 3 tasks. Both passed where they ran. Both failed on the clean image.

One lab's model behaving badly is the least interesting reading of this. Two different models found two different ways to make the tests agree with them, and a third class of shortcut, reading the answer off the network, showed up in the same audit. The pattern is structural: when the thing being graded controls the room the grading happens in, a capable enough agent will eventually rearrange the room. The interesting part is how the V2 harness is built to stop that, what it still cannot stop, and what the same design says about your own agents when they tell you "all tests pass."

What an in-place grade actually measures

A SWE-Bench style task is a real repository at a real commit, an issue description, and a set of hidden tests that the upstream fix made pass. The agent gets a container with the repository checked out, a shell, and a budget. When it stops, the harness runs the tests and records pass or fail.

The weak point is the phrase "runs the tests." In the simplest harness the tests run in the same container the agent just spent an hour inside. Everything the agent did is still there: the edited source files, but also any file it wrote outside the repository, any package it installed or modified, any cache it warmed, any stash or ref it created. The grade answers the question "do the tests pass in this particular machine state," and the agent authored that machine state.

Go makes the gap easy to see because Go is careful about dependencies. A module's go.mod names its dependencies and versions. go.sum records a cryptographic hash of each dependency's content, and the toolchain refuses to build if downloaded code does not match the recorded hash. Downloaded modules are extracted into the module cache ($GOMODCACHE, by default under $GOPATH/pkg/mod), and builds read them from there. The repository diff only shows go.mod and go.sum. The cache is outside the repository entirely.

So there are two ways to change what code a Go test actually executes without the change showing up as an honest fix. You can edit the dependency's source in the cache, which the diff never shows. Or you can point the module at a version whose content suits you and write a go.sum line that vouches for it. Scale's post describes exactly those two: one frontier model "forging a Go module version and its checksum, and another editing dependencies directly in the module cache. Both passed in place and failed on a clean image." In the agent's container the forged hash matched the locally doctored module, so the build went through. On a pristine image the doctored content does not exist, and a hash with nothing honest behind it cannot be satisfied.

The V2 fix is architectural: the grader no longer runs in the sandbox the agent used. The harness captures the agent's diff, and only the diff, then applies it to a fresh image and grades there. Anything the agent changed outside the diff evaporates. Scale publishes both grades per task, so a reader can see where they disagree.

The shortcuts are older than this benchmark

None of these moves is new in kind. In 2025 the SWE-bench team opened issue #465, "Repo State Loopholes During Agentic Evaluation", after finding trajectories where an agent ran git log --all and read future commits that contained the fix. The checked-out repository still carried history past the task's starting commit. A team member described it on Hacker News as a classic bug in code they believed already hid future history, and patched it.

A few months earlier METR had published "Recent Frontier Models Are Reward Hacking", which counted reward hacking attempts in 39 of 128 runs, 30.4%, on its RE-Bench tasks. The examples read like the V2 incidents with different nouns: a model that walked the Python call stack to find the grader's precomputed answer, and a model that replaced an evaluation function with a stub that marked every submission correct.

V2 answers each of these with a specific removal. Per Scale's post, images are sanitized of "the fixing commit, stray refs, stashes, hooks, and test files," agent edits to fixtures and snapshots are reverted, and stale Python bytecode is purged before tests are applied. The network got the same treatment after an audit: during an earlier open-network run, 32 of 642 trajectories called code hosts and 4 retrieved the fixing commit's SHA. The agent phase now reaches only the model endpoint, with web tools disabled.

Read the list as a map of where information about the answer can live: in history, on the network, in the environment, in the grader's own process. A benchmark is only as clean as the least sanitized of those four places.

The grader had bugs too

The same audit found problems that had nothing to do with agents. Of the original 731 public tasks, Scale removed 89 as invalid and found that 69 tasks had instructions that contradicted the tests grading them. It corrected the text only, then had an expert solve each one blind from the instruction alone. According to the post, the work took 1,897 hours from 23 contracted engineers.

The most instructive catch was Scale's own regression. V2 runs a two-sided gate before release: every task must pass with the reference patch and fail with the empty patch. That gate caught "a Jest parser fix that silently broke 23 element-web tasks." Without it, a perfect fix to any of those 23 tasks would have scored as a failure, and nobody would have seen why. The two-sided check is cheap and catches both directions of error: a task whose tests pass with no patch at all measures nothing, and a task whose reference fix fails measures noise.

An independent group reached the same diagnosis in the same weeks. "SWE-Bench Pro Verified" (Zheng et al., v1 September 8, v2 September 16) names two sources of unreliability in the original benchmark, reward hacking through leaked gold solutions or hidden evaluation information, and task quality issues such as misleading problem statements and improperly scoped tests. Its re-evaluation found that "some models perform substantially worse than previously evaluated." Two teams, working separately, landed on leakage plus bad tasks.

The number that survives the cleanup

Here is the part most coverage skipped. After all the hardening, the public split still looks almost saturated. Scale's post lists resolved counts on the 642 public tasks and on a 272-task private split that is never published: Claude Opus 5 at 638 of 642 public and 222 of 272 private, Kimi K3 at 627 and 214, GLM-5.3 at 614 and 211, Gemini 3.8 Flash at 609 and 211, Inkling at 577 and 184.

Converted to rates, Opus 5 goes from 99.4% on public tasks to 81.6% on private ones, the 17.8 point gap Scale calls out. Scale is explicit about why it thinks this is not evaluation-time cheating: the public runs were network-locked and audited, "with no successful retrieval from code hosts or module proxies." It attributes the gap to training-time exposure, since these are public repositories the models saw during training. "The private set is the only clean measurement of that, and it is why we maintain one."

Two things fall out of the figure. First, the public split spreads five models across about 9.5 points, while the private split spreads them across about 14, so the clean set separates models better than the contaminated one does. Second, every model loses between roughly 17 and 22 points going private. The cheats V2 caught were individual tasks. Training exposure moves the whole board.

That ordering of magnitudes matters for anyone reading launch posts. A forged checksum is a vivid story and worth fixing, but a benchmark that a model has partly seen during training inflates the headline far more, and no amount of sandboxing at evaluation time removes it. Only fresh tasks do.

What V2 says it still cannot stop

Scale lists two residuals in plain words: "The model endpoint is a trusted relay, and code inside a patch (conftest.py, a go.mod replace, a Makefile target) is still executed by the verifier."

The second one deserves a close read, because it is the pristine regrade's blind spot. The regrade discards everything outside the diff, but the diff itself runs. Pytest automatically imports any conftest.py it finds on the way to a test, so a patch that adds one can change how tests are collected or reported. A replace directive in go.mod tells the Go toolchain to build a dependency from a path the patch controls. A Makefile target is shell. Each of these is legitimate in a real fix, which is why they cannot simply be banned. They have to be reviewed like code, because they are code, and they execute inside the grader.

The first residual is quieter. The agent can reach the model endpoint and nothing else, which means the benchmark trusts whatever sits behind that endpoint not to be a channel for task answers. For a public leaderboard that runs labs' own endpoints, that is a trust assumption with no control behind it.

Scale also notes a failure in the other direction: on many tasks the Harbor evaluation framework failed to install an agent harness, and the task was recorded as a false negative. The two-sided gate checks tasks. It does not check the machinery around them.

The harness moves the bill more than the score

V2 shipped with one more experiment, run by Reflection: the same models under three open harnesses, mini-swe-agent, Pi, and OpenCode. The finding was that "the change in accuracy is very small for Kimi-K3 and GLM-5.3, the swing in cost is bigger," with Kimi K3 under Pi spending the fewest tokens per solved task. The post's summary line is "Pick the harness for the bill, not the score."

A separate paper from the same week points the same way from the other side. "An Empirical Study of Harness Design for Coding Agents" (Fan et al., September 17) holds the agent loop fixed and varies planning, action space, and context management across 176 matched settings on four models, using SWE-Bench Verified and Terminal-Bench 2.1. Context management mostly pays off by preventing overflow when the window is tight. Planning acts as an accuracy scaffold for weaker models and a cost saver for stronger ones. Bash-capable models do well with a bash-only interface at substantially lower cost, while predefined tools help models with weaker bash skills.

Put the two results next to the cheating incidents and a consistent picture forms. For strong models on a cleaned benchmark, the harness mostly decides what the run costs. What decides whether the score means anything is the grading design: where the tests run, what the agent could reach, and whether the tasks were ever seen before. I made the harness-side version of this argument in The Harness, Not the Model. This is the evaluation-side version.

Your agent's "all tests pass" is an in-place grade

The lesson carries straight into daily work. When a coding agent finishes a task on your laptop and reports green tests, it is reporting an in-place grade. It ran the tests in the environment it just modified. Most of the time that is fine. The V2 incidents show that a capable model under pressure to pass will sometimes make the environment agree with it instead of the code, and nothing in the final message tells you which happened.

The V2 design translates into a short checklist that works for private evals and for agent pull requests alike:

• Grade the diff, not the machine. Capture the patch, apply it to a fresh checkout or container, and run tests there. Fresh CI runners already do this for pull requests. Local agent loops usually do not.
• Strip the answer from the environment. Future history, stashes, refs, hooks, and cached artifacts should not exist where the agent works. Lock the network to what the task needs.
• Gate every task in both directions. Reference patch passes, empty patch fails, before any model sees it. This catches your own harness regressions, like the Jest parser bug.
• Review executable config as code. conftest.py, go.mod replace lines, build files, and CI config in an agent's diff run inside your grader. Flag them.
• Keep a split nobody trains on. Tasks that never leave your organization are the only way to measure fresh skill, as Scale's private set shows.
• Watch writes outside the repository. A trajectory that edits the module cache, site-packages, or test fixtures is worth reading even if it scored a pass.

I wrote in Stop Agents Building the Wrong Thing that the harness has to define what done means before the agent starts. SWE-Bench Pro V2 adds the other half: done has to be checked somewhere the agent never stood.