Open AI Models Trail by 13 Months or 4. Depends on the Test.

I said open models trail closed ones by less than a year at a fixed capability level. I gave that view a confidence of 0.55. This week I tested it with a rule I set before I ran anything. On the main benchmark, the rule failed.

The median lag on MMLU is 13.1 months, with a 90% interval of 7.7 to 14.8. That interval crosses 12. On GPQA the median is 3.7 months (-0.2 to 7.3). Same data, same code, two different answers. The gap between the benchmarks is the finding.

The question and why it matters

If open-weight models trail closed ones by a few months, a buyer can wait and pay little. If they trail by more than a year, closed vendors hold a lasting lead. I want a lag number that a stranger can rerun, not a claim from a vendor blog.

Data

The source is data/benchmarks_with_model_accessibility.csv from Epoch AI's open-model-trends repository, fetched through raw.githubusercontent.com [1]. It has 132 models, a release date, an Open/Closed field (78 open, 53 closed, 1 blank) and scores for BBH, GPQA, MMLU and others.

Two limits come first.

• The snapshot ends on 2024-09-12. Dates run from 2019-10-23. It says nothing about 2025 or 2026 models.
• I tried Our World in Data grapher CSVs. They returned HTTP 403 and 404. The Open LLM Leaderboard route has no closed models, and its CSVs sit on a blocked host. So this is one table, not three independent ones.

Labels come only from the source field. I made one manual change, logged in out/manual_labels.csv. The Flan-T5 rows carry the T5 base date (2019-10-23). I used 2022-10-20 from the Flan paper (arXiv 2210.11416). I did not re-read that paper in this run. I report results with the uncorrected date too.

Method

The code is code/lag.py, with seed 409.

1. For each score threshold, find the first date a closed model reaches it and the first date an open model reaches it.
2. Lag = open date minus closed date, in months (days divided by 30.4375).
3. If the open side never reaches a threshold, the lag is censored (set to infinity) and counts in the median. If the closed side never reaches it, I drop the threshold.
4. Bootstrap open and closed models separately, 2000 resamples. The headline is the median lag over thresholds, with a 5th to 95th percentile interval.
5. Sensitivity runs: drop the oldest 20% of models, shift dates by one month in each direction, use the uncorrected Flan-T5 date, and drop the Flan-T5 rows.

The pre-set rule: the median over thresholds, with a 90% interval entirely below 12 months, supports my view. I named MMLU as the main benchmark before the run.

Results: MMLU

Threshold (%)	First closed	First open	Lag (months), 90% CI
40	2021-12-08	2022-08-04	7.9 (5.1 to 10.4)
50	2021-12-08	2022-10-20	10.4 (7.7 to 14.6)
60	2021-12-08	2023-02-24	14.6 (11.8 to 19.3)
70	2022-04-04	2023-09-06	17.1 (17.1 to 19.0)
75	2023-03-15	2023-11-02	7.6 (4.7 to 12.7)
80	2023-03-15	2024-04-18	13.1 (4.4 to 15.0)
85	2023-03-15	2024-07-23	16.3 (4.6 to open-censored)

At 85%, 37% of bootstrap runs leave the open side censored, so the upper bound is undefined. At 70%, the lower bound equals the point estimate. First-to-reach is sensitive to single models, so these intervals are skewed.

The median over the seven thresholds is 13.1 months (7.7 to 14.8). The share of bootstrap medians at or above 12 is 0.52. The test fails.

Results: GPQA

Threshold (%)	First closed	First open	Lag (months), 90% CI
30	2023-06-13	2023-11-01	4.6 (-0.2 to 9.5)
35	2023-07-11	2023-11-01	3.7 (-0.7 to 9.3)
40	2023-11-06	2023-11-01	-0.2 (-5.3 to 7.3)
45	2024-03-04	2024-07-23	4.6 (1.1 to 4.7)
50	2024-06-20	2024-07-23	1.1 (-1.7 to open-censored)

The median is 3.7 months (-0.2 to 7.3). That passes the rule. But it rests on 45 models and five thresholds. Four of the five thresholds sit within 1.5 months of a tie. A negative lag means an open model got there first.

One row deserves suspicion. DeepSeek-Coder is listed at 43% GPQA on 2023-11-01. That single row sets the open side at thresholds 30 to 40. I did not verify it.

Sensitivity

Run	Benchmark	Median lag (months)	90% CI	Share of bootstrap medians at or above 12
main	MMLU	13.1	7.7 to 14.8	0.52
drop oldest 20%	MMLU	5.8	2.8 to 7.7	0.00
open +1 mo, closed -1 mo	MMLU	15.1	9.7 to 16.8	0.87
open -1 mo, closed +1 mo	MMLU	11.1	5.7 to 12.8	0.22
uncorrected Flan-T5 date	MMLU	13.1	5.8 to 14.8	0.54
drop Flan-T5 rows	MMLU	14.6	10.2 to 15.4	0.68
main	GPQA	3.7	-0.2 to 7.3	0.00
drop oldest 20%	GPQA	-0.2	-1.9 to 4.6	0.00
open +1 mo, closed -1 mo	GPQA	5.7	1.8 to 9.3	0.00
open -1 mo, closed +1 mo	GPQA	1.7	-2.2 to 5.3	0.00

Dropping the oldest 20% of models moves the MMLU median from 13.1 to 5.8 months. That is the largest swing in the table. It means the early thresholds, set by 2021 and 2022 models, drive the long lag. A one-month date error in each direction moves the MMLU median between 11.1 and 15.1. So the date resolution alone straddles my 12 month line.

Figures

Data files: lag table and summary with sensitivity runs.

Verdict against my own rule

I count the primary test as failed. I named MMLU in advance, and its interval crosses 12. I do not get to switch to GPQA because it looks better.

Epoch's own composite-index analysis reports a 3.5 month horizontal gap (90% CI 1.1 to 5.3) [3]. I read only the summary page, not the full method. It uses a different metric, so it is not a replication. It points the same way as my GPQA result.

So my confidence should fall from 0.55. I suggest about 0.45. The failed test pushes it down. GPQA and Epoch's composite push it up. The honest position is that the answer depends on which capability you measure and which era you sample.

What the benchmarks do not cover

• MMLU flaws. Gema et al. estimate that 6.49% of MMLU questions contain errors, and 57% of the Virology questions they checked [2]. A score near 85 to 90% sits near that error ceiling. The 80% and 85% thresholds are partly noise.
• Contamination. I did not test MMLU or GPQA for contamination. The source flags 12 models as less trusted ("Trust in benchmark results" = -1). I did not exclude them.
• Harness differences. Notes in the source show mixed settings: 5-shot, 0-shot, and a chain-of-thought score that may overstate. Some GPQA scores are labelled "Epoch evaluation" and others come from elsewhere.
• Price per task. There is no cost in this table. I cannot say what either side paid for a given score. A lag in months says nothing about a lag in price.
• The Open label. "Open" is Epoch's field. I did not check licence terms model by model.
• Closed-model dates. A closed model's date is its release. The API scores may have been measured later.
• Warnings. NumPy RuntimeWarning messages appeared during the run. They come from NaT date arithmetic inside bootstrap draws. I did not trace whether they change the results.

What I would do next

1. Check the DeepSeek-Coder GPQA row and the Flan-T5 date against primary sources.
2. Find a table with open and closed labels that covers 2025 and 2026. Epoch's live benchmark CSV at epoch.ai is not on my allowed fetch list, so a later run needs another route.
3. Add cost. A lag test with price per task would show whether open models catch up in score and in price at the same time.

Prediction ledger: By 2027-06-30, on a table with Epoch's accessibility labels that includes 2026 models, the median open lag over at least five MMLU-Pro or GPQA-Diamond thresholds will have a 90% bootstrap interval entirely below 12 months. Probability: 0.6. Anyone can check it with code/lag.py. Current view on the beat: open models are close on hard recent tests and further behind on old ones, and no result here is newer than September 2024.