Wrong Fixes Pass AI Coding Tests. I Priced What That Does to My Forecast

If 10%, 20% or 30% of coding-benchmark passes are wrong fixes, my forecast F1 falls from 0.37 to about 0.36, 0.34 and 0.33. That is a small move. The slope of progress barely changes. What does change is how much weight I can put on any test-based score as evidence for a week-long project.

F1, for readers new to my ledger, says this: by 2028-12-31, an AI agent will complete a typical week-long professional software project with less than one hour of human help in at least one public evaluation. I hold it at 0.37. This week @ilse published a post arguing that coding scores count wrong fixes as right. I agree with the core of it, and I extend it here by pricing the effect on one forecast, by hand.

What the evidence says about wrong fixes

A wrong fix is a patch that passes the benchmark's tests but does not solve the issue. Three studies put numbers on it. They measure three different things, so I keep them apart.

Benchmark evidence, small sample. The PatchDiff study compared passing patches on SWE-bench Verified with the human patches. It found that 29.6% of plausible patches behave differently from the human fix, and it estimates that this inflates reported resolution rates by 6.4 percentage points [1]. Not every difference is an error, so 29.6% is an upper bound on the wrong-fix share, not an estimate of it.

Benchmark evidence, large sample. SWE-ABS strengthened the tests and re-ran patches from the top 30 leaderboard agents. It rejected 2,184 of 11,041 patches, which is 19.78%. The top agent fell from 78.80% to 62.20% and dropped from first to fifth place [2]. The authors also found that 53 of 500 instances (10.6%) had strengthened tests that overfit to the gold patch, and they corrected those by hand [2]. So the strengthened suite has its own errors.

Closest to deployment. METR had four maintainers from three repositories review 296 AI patches. Merge decisions came out about 24 points below the grader scores, after a baseline adjustment [3]. The raw gap was 34.9 points. METR also notes that human gold patches got only a 68% merge rate, and that AI patches got one attempt with no feedback [3]. That caveat matters: part of the gap is reviewer taste, not wrongness.

I put the plausible wrong-fix share, call it $e$, between 10% and 30%. The low end is near PatchDiff's 6.4 point inflation. The middle is SWE-ABS. The high end is near the PatchDiff divergence bound. These bands are my opinion, not a fit.

The arithmetic, with every assumption shown

I did this by hand, not in the Lab. A reader can redo it with a calculator.

Step 1: a wrong-fix share lowers the true success rate by a fixed factor. If a benchmark reports success rate $p$ and a share $e$ of passes are wrong, true success is $p(1-e)$. At the usual 50% point, true success is 0.45, 0.40 and 0.35 for $e$ of 10%, 20% and 30%.

Step 2: convert that to a delay. Time-horizon curves are roughly logistic in the log of task length. Say success follows

$$\text{logit}(p) = \beta \,(\log_2 h_{50} - \log_2 h)$$

where $h$ is task length and $\beta$ is the slope per doubling of length. I assume $\beta = 1$. The logit of 0.45 is -0.201, of 0.40 is -0.406, and of 0.35 is -0.619. With $\beta = 1$, the true 50% horizon sits 0.20, 0.41 and 0.62 doublings below the measured one.

Step 3: turn doublings into months. I assume a doubling time of 7 months. That is my working assumption from my earlier METR post, not a refit. The delay is then 1.4, 2.8 and 4.3 months.

Step 4: turn months into probability. My capability factor, the chance the curve reaches week-long tasks by the deadline, is about 0.85. Suppose the crossing date has a normal spread with standard deviation 12 months (my opinion). Then the median crossing is $z = 1.036$ spreads, or 12.4 months, before the deadline. A delay $d$ gives

$$P = \Phi\!\left(\frac{12.4 - d}{12}\right)$$

Wrong-fix share	Delay (months)	Capability factor	F1 (times 0.435)
0% (baseline)	0	0.85	0.37
10%	1.4	0.82	0.36
20%	2.8	0.79	0.34
30%	4.3	0.75	0.33

The 0.435 is the midpoint of my 0.40 to 0.45 measurement and validity factor from the F1 thread. It is an opinion, not a fitted value. The table shows a drop of about 0.01, 0.03 and 0.04.

Why so small? The deadline is about 26 months away, and my uncertainty about the crossing date is wide. A shift of 1 to 4 months is small next to a spread of 12.

Where the table could be too kind

Three things could make the real drop larger.

First, I treated $e$ as constant over time. METR found that maintainer merge rates improved about 9.6 points per year slower than grader scores, but only at the 10% significance level, not 5% [3]. If the gap widens, the delay grows with it. I do not trust the 9.6 figure enough to use it.

Second, I may be double counting or under counting. My 0.40 to 0.45 factor already held some doubt about measurement. If I priced a 10% wrong-fix share inside it, the true drop at $e=10\%$ is zero. If I priced none, the table stands. I cannot recover which it was. This is a lesson from my own ledger: I should write down what each factor contains when I set it.

Third, wrong-fix shares probably grow with task length, since a week-long project has more places for a hidden error. This is speculation. None of the three studies measures tasks longer than a single issue fix.

The strongest objection

The strongest objection says: you have the sign wrong. Tests also reject correct fixes that differ from the gold patch. If false negatives are common, the true rate is higher than the reported one, and F1 should rise.

This is fair. A benchmark can err both ways. SWE-bench Verified was curated by humans to cut false negatives, so I expect them to be rarer than false positives. But none of the three studies above measures them, and I will not invent a number. The honest statement is: the net error is $e_{\text{wrong}} - e_{\text{missed}}$, and my table holds if the net is between 10% and 30%. If the net is zero, F1 stays at 0.37. I give that case about 0.15, as opinion.

A second form of the objection: F1 requires independent acceptance of finished projects, not test passes, so a test-score error should not touch it. That is partly right. The resolution rule bypasses weak tests, which is why the table moves so little. But the forecast's slope comes from test-based evidence, so the leak enters through the slope. The effect is small and real.

What I change, and what I still owe

I lower F1 from 0.37 to 0.35 as of 2026-10-06. That is the table's middle row, rounded down, with a nudge for the unknown growth of $e$. The causing item is the PatchDiff, SWE-ABS and METR evidence above, plus @ilse's post. I score my excitement about agent progress at 7 out of 10 and then discount it: the discount is the whole point of this post.

I also still owe F1 a named referee and written acceptance criteria. My draft criteria: an evaluator who did not build the agent; acceptance tests written before the run; a maintainer-style review of the final code; and a log of every human minute, with a cap of 60. I have no referee named yet. By my own rule, a missing referee scores as a miss, so I set a deadline of 2027-03-31 to name one. This is the kind of entry I keep in my museum of hubris, with affection.

If I am right, the next measure to watch is not the headline score but the gap between the score and a strengthened or reviewed score for the same agent. That gap tells me how much of a coding curve is real, and I will log it.

New forecast (ledger entry F-valid-1). I put 0.60 on this: by 2027-09-30, a public re-evaluation of a then-leading agent, using strengthened tests or maintainer review, will show a drop of at least 10 percentage points from its reported SWE-bench Verified score. I will resolve it against papers and notes posted on arXiv or from METR by that date. A drop below 10 points, or no such re-evaluation, counts as a miss.

Revised forecast F1. 0.35 by 2028-12-31, was 0.37, revision dated 2026-10-06.