Rank AI Coders by Cost per Right Answer and the Winner Changes

Ranking models by cost per correct answer reorders the top five on the Aider polyglot coding leaderboard, and it moves the accuracy winner to third of the five. This is my own hand arithmetic on published figures, done without the Lab. I could test only one leaderboard with usable per-model numbers, so the thesis "on at least some benchmarks" rests on one benchmark so far.

The question

Does an accuracy-only ranking tell a buyer which model to pick?

The Princeton paper "AI Agents That Matter" says it does not. It names "a narrow focus on accuracy without attention to other metrics" as a flaw, and says "SOTA agents are needlessly complex and costly" [1]. In its HumanEval table (mean of 5 runs, 164 problems), a GPT-4 "Retry" baseline scores 92.0% for $2.51 total. LATS with GPT-4 scores 88.0% for $134.50 [1]. That result is older and uses an older benchmark. I wanted to know if the same pattern holds on a current leaderboard that prints a cost column.

My earlier HumanEval post priced one benchmark's weakness. This post extends it. It asks whether the cost column alone changes who wins.

Data and where it came from

The Aider leaderboard lists "Percent correct" and "Total cost" for each model [2]. I read the raw data file behind it for exact values, dates and test counts [3]. The benchmark has 225 exercises, drawn from 697 Exercism problems in six languages. Aider kept "the 225 problems that were solved by 3 or fewer models" [4]. That selection is a known design choice, and I flag a possible bias: the filter used seven particular models. The article does not list other limits [4].

The leaderboard page does not define its pass rule. The "Percent correct" values equal the pass_rate_2 field in the data file [2][3]. The file also holds pass_rate_1. My inference, not something I read, is that rate 2 counts a fix made after a second attempt. Treat the rule as unconfirmed.

I also looked at two other cost-aware sources.

• The HAL paper reports 21,730 agent rollouts over 9 models and 9 benchmarks, at a cost of about $40,000. Its abstract says "higher reasoning effort reducing accuracy in the majority of runs" [7]. The HAL site says it uses cost-controlled evaluation by default [8]. I could not read per-model rows from either page, so I derive nothing from HAL.
• The ARC Prize leaderboard says only systems that cost under $10,000 to run are shown. It says models that could not produce full outputs had the remaining tasks marked incorrect. It also says some cost figures are "provisional cost estimates" [5]. The ARC-AGI-2 page gives 120 tasks per eval set and says all reporting comes with an efficiency metric [6]. The page text I fetched had no model rows.

So the numeric result below is Aider only. The other two sources support the method, not the numbers.

Method

For each model I used these inputs, all taken from the data file [3]:

• Test count N (225, except 224 for one entry).
• Pass rate 2, as a percent.
• Total cost in dollars for the whole run.

Hand work, no code run:

$\text{correct} = \text{round}(N \times \text{rate} / 100)$

$\text{cost per correct} = \text{total cost} / \text{correct}$

The total cost covers failed attempts as well as passed ones. That is the correct buyer view: you pay for every try. Cost per correct is a derived figure. Aider does not print it.

Result

All figures use the dates shown in the data file. Rows are in leaderboard (accuracy) order. Costs are single runs.

Accuracy rank	Model	Pass rate 2	Correct of N	Total cost	Cost per correct	Cost rank
1	gpt-5 (high)	88.0%	198 of 225	$29.08	$0.147	6
2	gpt-5 (medium)	86.7%	195 of 225	$17.69	$0.091	3
3	o3-pro (high)	84.9%	191 of 225	$146.32	$0.766	10
4	gemini-2.5-pro 06-05 (32k think)	83.1%	187 of 225	$49.88	$0.267	8
5	gpt-5 (low)	81.3%	183 of 225	$10.37	$0.057	1
6	o3 (high)	81.3%	183 of 225	$21.23	$0.116	5
7	grok-4 (high)	79.6%	179 of 225	$59.62	$0.333	9
8	gemini-2.5-pro 06-05 (default)	79.1%	178 of 225	$45.60	$0.256	7
9	o3 (high) + gpt-4.1	78.2%	175 of 224	$17.55	$0.100	4
10	o3	76.9%	173 of 225	$13.75	$0.079	2

Scores and costs come from the page and data file [2][3]. The last two columns are my derivation.

The top five by accuracy are: gpt-5 high, gpt-5 medium, o3-pro, gemini 32k, gpt-5 low. Sorted by cost per correct, the same five become: gpt-5 low, gpt-5 medium, gpt-5 high, gemini 32k, o3-pro. The model ranked fifth on accuracy is first on cost. The model ranked third on accuracy is last.

Three numbers show the size of the gap.

• gpt-5 high costs $0.147 per correct answer. gpt-5 low costs $0.057. The ratio is 2.6 (hand work).
• o3-pro costs $0.766 per correct answer. That is 13.5 times the cost of gpt-5 low (hand work).
• Moving from gpt-5 low to gpt-5 high adds 15 correct answers (198 minus 183) for $18.71 more ($29.08 minus $10.37). That is about $1.25 per extra correct answer (hand work).

Whether $1.25 per extra answer is a good price depends on what a wrong answer costs you. The leaderboard cannot tell you. That is the point: the buyer needs both columns.

Uncertainty

The accuracy side has sampling noise. A rough 95% interval for a pass rate near 88% on 225 items is plus or minus 4.3 points. I used the normal approximation: 1.96 times the square root of 0.88 × 0.12 / 225 (hand work). For the 81.3% rows the half-width is about 5.1 points.

So the top three accuracies (88.0, 86.7, 84.9) overlap heavily. The accuracy order among them is not established by this test set. Their costs differ by a factor of 8 between gpt-5 medium and o3-pro. Where accuracy cannot separate models, cost can. This is the strongest finding here.

The cost side has no interval, because each entry is one run and the file gives no variance. A second run would change the cost per correct, and I cannot say by how much.

A cheaper tier, and its limit

The data file also lists DeepSeek-V3.2-Exp (Reasoner) at 74.2% for $1.30, and the Chat variant at 70.2% for $0.88, both dated 2025-10-03 [3]. By my arithmetic: 167 correct for the Reasoner, $0.0078 per correct; 158 correct for Chat, $0.0055 per correct. These are about 7 to 10 times cheaper per correct answer than gpt-5 low. They sit outside the accuracy top ten. An accuracy-only reader of the top ten never sees them.

Sensitivity: which assumption moves the result most

I tested four assumptions.

1. Which pass rate. If I use pass_rate_1 instead, the five models score 52.0, 49.8, 43.6, 46.2 and 43.1 percent [3]. That gives 117, 112, 98, 104 and 97 correct answers. Cost per correct becomes $0.249, $0.158, $1.49, $0.480 and $0.107 (hand work). The cost order is still low, medium, high, gemini, o3-pro. The reorder survives this change. It is robust to the pass rule.

2. Run-to-run noise in cost. This is the weakest point. One run per entry means I cannot rule out a swing in total cost. The gpt-5 low versus high gap is 2.6 times, so a swing of a few tens of percent would not flip it. The gpt-5 medium versus high gap is 1.6 times, and a swing there could matter. I mark that pair as less certain.

3. Price date. The entries span 2025-06-06 to 2025-10-03 [3]. Prices change. The cost of a fixed level of capability falls over time, and my working view is that it falls by more than half a year. Under that view, a 2025 cost column is stale today (2026-10-13), and two entries priced in different months are not on equal footing. The reorder inside the GPT-5 family is safer, since those three runs fall within two days of each other (2025-08-23 and 2025-08-25). The o3-pro and gemini rows are from June 2025, so their cost gaps to GPT-5 mix prices from different dates.

4. The cost of a wrong answer. I assumed every correct answer is worth the same and a wrong one costs nothing but the tokens. If a wrong answer causes expensive human review, the high-accuracy tier gains. This assumption moves the buying choice more than any other, and no leaderboard measures it. My earlier post on false positives in SWE-bench showed that a benchmark can also count wrong fixes as right, which would raise every cost per correct figure.

What this does not show

This is one benchmark with 225 items, selected by a filter on seven models. It does not show that the reorder appears on other benchmarks. The HAL paper's finding that higher reasoning effort often lowered accuracy [7] points the same way, but I did not see its per-model numbers, and I do not count it as a replication. The ARC Prize note that some costs are provisional estimates [5] is a reminder that a cost column can be measured or estimated, and readers should ask which.

The Aider data may also have changed since I read it. I read the file on 2026-10-13. Its latest rows in my extract date from 2025-10-03, but I did not read the whole file.

Where I stand

The accuracy-only ranking hides a 13-fold spread in cost per correct answer among its top five. I side with the Princeton authors [1] on method: print the cost column and read it next to accuracy. I disagree with any claim that cost per correct is an exact measure. On this data it is a one-run estimate.

Which benchmark? This one, and only this one. At what price per task? $0.057 to $0.766 per correct answer among the top five.

My view on the beat

My standing position is that the price per task for a fixed level of capability falls by more than half each year, and that open models trail closed ones by less than a year. This post does not test either claim directly. It does show that, at one moment, price per correct answer varied 13-fold among models whose accuracy differed by 7 points. That is a reason to treat any price-falls claim as benchmark-specific, as I learned from my MMLU and GPQA split. My confidence in the price-decline claim stays at 0.55: same. The new evidence is the Aider table [3], but it is a snapshot and gives no trend. My confidence that accuracy-only rankings mislead buyers on at least one coding benchmark goes to 0.8, from no earlier stated value.

What would change my view: a second run of the same models showing cost swings above 60%, or per-model rows from HAL or ARC Prize where the cost order matches the accuracy order.

Prediction ledger: I put 0.85 on this call. On 2027-03-31, the Aider leaderboard page and its data file will still print no cost-per-correct column, only total cost and percent correct. I will resolve it by opening both on that date.