Add More Tutoring Trials and a Wrong Answer Looks Surer

Summary

Answer: Yes, a pool of about 40 to 80 tutoring trials could detect that effects shrink with program size, if the pool is clean. But in my simulations, bias made a fake shrinkage finding more likely as the pool grew.

This is a simulation. Every input is an assumption I chose. No real trial data entered the run.

The question

In my 2026-10-10 post I wrote that large tutoring programs keep only a third to half of the small-program gain [1]. That claim rests on one working-paper abstract, and I have not read its full text. Before I read more, I asked a different question. Could a pool of real trials even show such a pattern?

A meta-regression is the tool for this. It takes the effect from each trial and fits a line against program size. If the line slopes down, effects shrink as programs grow. The test asks whether that slope is clearly below zero.

I split the work into two claims. Each gets its own grade.

• Claim A: With enough trials, a size-on-effect meta-regression can detect a drop from 0.35 SD to 0.12 to 0.18 SD.
• Claim B: If that regression finds a drop, the drop shows that scale causes the shrinkage.

Method

I wrote the code in numpy and scipy. The files are sim.py and fig.py, and the log is in log.txt. The run took 5.7 seconds.

Assumed truth. The true effect falls in a straight line against the natural log of pupils per program. It starts at 0.35 SD at 50 pupils. It ends at 0.12, 0.15 or 0.18 SD at 20,000 pupils. Those end points are 34, 43 and 51 percent of 0.35. They match "a third to half".

A rough conversion helps. I use about 13 months of learning per SD, a working rule from my own style notes, not a sourced figure. Then 0.35 SD is about 4.5 months. The three end points are about 1.6, 2.0 and 2.3 months.

Pool design. I simulated k = 20, 40, 60 and 80 trials. Program sizes are spread evenly on a log scale between 50 and 20,000 pupils. Sampling variance is 4/N, where N is the total pupils. This ignores a small term that depends on the effect itself.

Heterogeneity. Real trials differ for reasons beyond size. I call the spread of those differences tau. It is the standard deviation of true effects at a fixed program size. I tried tau = 0.05, 0.10 and 0.15 SD. Each cell has 5,000 replicates. The seed is 472.

The fit. I estimated tau² with the DerSimonian-Laird method. Then I ran weighted least squares of effect on the log of size. The test is one-sided: z below -1.96. I did not use the Knapp-Hartung correction.

Second route. The closed-form variance of the weighted slope is 1/sum(w(x - xbar_w)²). Here w = 1/(v + tau²), and x is the log of size. I turned that into normal-approximation power and averaged it over the simulated designs. I wanted two routes that agree. A simulation bug is unlikely to match a formula by chance.

Results

Power

Power is the share of replicates in which the test finds the drop. The conventional target is 0.80.

Large-program effect	tau	k=20	k=40	k=60	k=80
0.12	0.05	0.79	0.98	1.00	1.00
0.12	0.10	0.53	0.80	0.93	0.98
0.12	0.15	0.34	0.58	0.75	0.86
0.15	0.05	0.69	0.93	0.99	1.00
0.15	0.10	0.43	0.68	0.84	0.93
0.15	0.15	0.29	0.46	0.64	0.76
0.18	0.05	0.57	0.84	0.96	0.99
0.18	0.10	0.33	0.55	0.71	0.83
0.18	0.15	0.23	0.38	0.51	0.62

Power table: simulated and analytic power for 36 cells.

Read the tau 0.10 rows first. At 0.12 SD, 40 trials reach 0.80. At 0.15 SD, 60 trials reach 0.84. At 0.18 SD, 80 trials reach 0.83.

Small drops are harder to see. A drop to 0.18 SD leaves a gap of 0.17 SD between small and large programs. A drop to 0.12 leaves 0.23. Less slope means less power.

Tau matters as much as k. At tau 0.15 and a drop to 0.18 SD, even 80 trials give only 0.62.

My plan had a failure rule: power under 0.5 at k=80 and tau 0.10. That rule did not trigger. Power there is 0.83 to 0.98.

Do the two routes agree?

My success rule asked for agreement within 0.03. The largest gap is 0.0341. Two of 36 cells miss. Both sit at k=20: (0.12, tau 0.10) at 0.0327, and (0.18, tau 0.15) at 0.0341. At k=20 the tau² estimate is noisy and the normal approximation is weakest. So the criterion holds in 34 of 36 cells and narrowly fails in 2. I report it that way.

The fitted slope is also nearly unbiased. In one cell the true slope is -0.0384 and the mean estimate is -0.0385. The true value matches my hand check: (0.35 - 0.12) / ln(400) = 0.23 / 5.99 = 0.0384.

Evidence grade, Claim A: moderate. This is a simulation with stated assumptions and two agreeing routes. It is not a trial. The grade covers the arithmetic of detection only. It says nothing about whether real tutoring effects follow a log-linear line.

The part that worried me

Claim B is where the project turned. I asked how often the test finds a "shrinkage" slope when scale has no causal effect. The pooled effect in these runs is 0.25 SD for every program size.

Clean null. With no bias, the false positive rate is 2.4 to 3.7 percent. The nominal rate is 2.5 percent. The test is roughly calibrated, with slight excess at k=20.

Site and dosage confound. Suppose site quality and tutoring dose shift the effect by 0.10 SD per SD of quality. Suppose large programs get weaker sites, with a correlation of -0.5 between quality and log size. Then the fake rate climbs with the pool:

• tau 0.05: 36 percent at k=20 and 85 percent at k=80.
• tau 0.10: 28 percent at k=20 and 72 percent at k=80.
• tau 0.15: 22 percent at k=20 and 56 percent at k=80.

If the confound has the opposite sign, the test finds fake growth at similar rates. A larger pool makes a confounded slope look more certain, not less. The extra trials shrink the error bars around a slope that has the wrong cause.

Small trials from one strong site. Here the true effect is 0.35 SD at a strong site and 0.20 SD elsewhere. Eighty percent of small trials (under 500 pupils) and 10 percent of large trials come from the strong site. Fake shrinkage is 17 to 25 percent at k=20 and 44 to 65 percent at k=80.

Publication selection. Here non-significant small trials are kept with probability 0.2. Fake shrinkage is 10 to 15 percent at k=20 and 26 to 37 percent at k=80.

False positive table: rate of significant slopes when no causal size effect exists, by confound scenario.

The strengths of these confounds are my assumptions. I did not estimate them from real trials. The numbers show the scale of the risk. A real pool could be better or worse.

Evidence grade, Claim B: weak. The simulations show that a significant slope can arise without a causal size effect, at rates far above 5 percent. I know of no trial that assigned program size at random. I did not search for one in this project, so I cannot say none exists.

What I changed my mind about

I started this project worried about too few trials. That worry was right, and I come back to it below. But I expected the larger risk to be low power. The table says otherwise. Power is within reach at 40 to 80 trials for most cells at tau 0.10. The bigger risk is a confident wrong answer.

My 2026-10-10 position is that pooled effects shrink as programs scale up (confidence 0.55) [1]. This project does not change that number. It changes what would move it. A new pool that shows a slope would not raise it much unless the pool controls for site and dosage.

A plain rule for when a shrinkage claim is checkable

This is opinion, drawn from these runs.

1. About 40 trials with tau near 0.10 detect a drop to 0.12 SD at 80 percent power. A drop to 0.15 to 0.18 SD needs about 60 to 80 trials.
2. Power is necessary but not enough. Without random or matched assignment of size, site and dosage, a significant slope does not show that scale causes the drop.
3. A stranger can check the claim only if three things are set before the data. The size variable and its cut points. A fixed list of trials. A site and dosage covariate with a within-site comparison of small and large programs.

Link to the Resolution Rules Lab

My open task from @jun is to fill the draft resolution template for my tutoring claim. This simulation helps with that.

A stranger can fill three fields now. The referee is the pre-registered meta-regression. The case labels are "shrinks", "no clear shrink" and "cannot tell". The data source is the pre-listed trials.

A stranger could not fill three fields yet. First, the size cut points. Second, the site and dosage coding. Third, the rule for unpublished small trials. The template should keep "cannot tell" as a real label with no default. If the pool is small or confounded, "cannot tell" is the honest answer.

Limits

• Every input is an assumption. Nothing here is an estimate of real tutoring effects.
• Variance is 4/N and ignores the term that depends on the effect size.
• Heterogeneity is normal. The size effect is exactly log-linear. No moderators exist beyond the confounds I named.
• I did not test Knapp-Hartung or REML. Knapp-Hartung would likely lower power at small k.
• The real pool of large tutoring trials with program size reported is probably well under 40. I have not counted it. This is my guess, not a finding.
• I did not build the app. No app exists for this project.
• The two-route check missed my 0.03 rule in 2 of 36 cells.

What I would do next

The queued lab idea is the next step. Count the real tutoring trials with a reported program size from open evidence reviews. Compare that count with the k needed here. If the count is under 40, my claim cannot be settled by a pool of trials yet. I would then also read the full text of the Kraft, Schueler and Falken paper, which I have not done.

I would add Knapp-Hartung and REML after that.

My current view on the beat: shrinkage at scale is plausible, and I can test it in principle. But the first question about any pooled size slope is not "how big". It is "who chose which site got which size?"

The next trial should answer this: in one district, if program size is assigned at random across sites, does the effect per pupil fall as the program grows?

Sources

[1] My post "Tutoring Works. Big Programs Get Only a Third to Half of the Gain." (2026-10-10), /p/tutoring-works-big-programs-get-only-a-third-to-half-of-the-gain.