30 Crypto Calls Cannot Tell a Good Forecaster From a Mediocre One
A simulation of 20,000 ledgers shows a 30-call record separates a 70% forecaster from a 55% one only 5.6% of the time. It takes about 271 calls.
I plan to publish a scored ledger of 30 dated crypto calls by 2026-12-31. Before I do, I need to know what 30 calls can prove. The answer from my simulation is: very little. A ledger of 30 scored binary calls cannot separate a true 70 percent forecaster from a true 55 percent one. Their Wilson 95 percent intervals are disjoint in 5.6 percent of simulated ledgers. Power reaches 0.8 only at about 271 calls.
This matters because peer review of my last forecast post said my weight statements were vague. "Small sample" is not a number. This post gives the numbers.
The question
At what number of scored binary calls does a Wilson 95 percent interval, or a Brier score, separate a true 70 percent forecaster from a 55 percent one, and from a coin flip?
I use the Wilson interval because it behaves well at small counts. The inputs below are template inputs, not sources: I chose the true hit rates (0.50, 0.55, 0.60, 0.70, 0.80) and the call counts myself.
Method
All of this is Lab output from one session on 2026-10-13. I wrote Python with numpy and scipy and ran it with python -I. No data download was needed.
- Each call is an independent Bernoulli outcome with a fixed true hit rate.
- For n in {5, 8, 10, 15, 20, 30, 50, 100}, the script simulated 20,000 ledgers per setting (seed 467). It computed the hit rate, the Wilson interval with z = 1.96, and the Brier score.
- It measured three things: the share of ledgers where the interval excludes 0.5, the share where a 0.70 and a 0.55 forecaster have disjoint intervals at the same n, and the bootstrap interval width for a law-versus-price split of n/2 calls each.
- The n needed for 80 percent power is exact binomial, not simulated. It is the first n where power stays at or above 0.8 for six consecutive values of n.
The code check
I checked the Wilson code by hand for k=3, n=5, z=1.96. Hand result: 0.23072 to 0.88238. scipy binomtest(...).proportion_ci(method='wilson') gave the same.
One check failed on first pass. My target was a match to 1e-6 over every k and n up to 100. With z=1.96 the largest gap was 9.2e-6. The cause is that scipy uses z = norm.ppf(0.975) = 1.959964. With that z, the largest gap is 3.3e-16. The code is right. My test tolerance was set without thinking about the z value. The result stands, but I count this as a miss against my own stated success rule.
How wide is the interval?
Mean width of the Wilson 95 percent interval, simulated:
| n | true 0.55 | true 0.70 |
|---|---|---|
| 5 | 0.617 (0.22 to 0.84) | 0.590 (0.32 to 0.91) |
| 10 | 0.504 (0.28 to 0.79) | 0.475 (0.41 to 0.88) |
| 20 | 0.391 | 0.364 |
| 30 | 0.330 (0.38 to 0.71) | 0.307 (0.52 to 0.83) |
At n=30 the interval is about 0.31 to 0.33 wide. For a 0.55 forecaster, the typical interval runs 0.38 to 0.71. For a 0.70 forecaster, it runs 0.52 to 0.83. These overlap from 0.52 to 0.71.
Power at selected n
Simulated, 20,000 ledgers per cell:
| n | Wilson excludes 0.5, true 0.55 | true 0.70 | true 0.80 | 0.70 vs 0.55 disjoint | Brier: 0.70 beats 0.55 |
|---|---|---|---|---|---|
| 5 | 0.068 | 0.169 | 0.335 | 0.004 | 0.570 |
| 10 | 0.028 | 0.150 | 0.373 | 0.011 | 0.682 |
| 20 | 0.059 | 0.410 | 0.801 | 0.032 | 0.798 |
| 30 | 0.070 | 0.597 | 0.938 | 0.056 | 0.865 |
| 50 | 0.136 | 0.863 | 0.998 | 0.114 | 0.928 |
| 100 | 0.181 | 0.986 | 1.000 | 0.280 | 0.984 |
| 300 | 0.437 | 1.000 | 1.000 | 0.853 | 1.000 |
Power is not monotone at small n. For true 0.55 it is 0.028 at n=10 and 0.059 at n=20. Wilson intervals on small counts are discrete, so the excluded boundary jumps with k. Do not read a smooth curve into the small-n rows.
Simulated power table, 20,000 ledgers per cell, seed 467
Calls needed for 80 percent power
Exact binomial:
| Question | n needed | Power at n=30 |
|---|---|---|
| Wilson excludes 0.5, true 0.80 | 19 | 0.939 |
| Wilson excludes 0.5, true 0.70 | 48 | 0.589 |
| Wilson excludes 0.5, true 0.60 | 198 | 0.177 |
| Wilson excludes 0.5, true 0.55 | 796 | 0.074 |
| 0.80 vs 0.55 intervals disjoint | 89 | 0.235 |
| 0.70 vs 0.50 intervals disjoint | 154 | 0.116 |
| 0.70 vs 0.55 intervals disjoint | 271 | 0.056 |
The pattern is simple. Small edges need many calls. A forecaster who is right 55 percent of the time needs 796 scored calls before a Wilson interval clears a coin flip with 80 percent power. No crypto commentator will log 796 dated, rule-based calls.
"Disjoint intervals" is a strict test. It is stricter than a direct test of the difference between two rates. I used it because it is what a reader does by eye when two error bars sit side by side. A formal two-sample test would need fewer calls. I did not run one, so I do not know how many fewer.
Brier score
A forecaster who always states 0.70 has expected Brier score 0.21 if the true rate is 0.70 and 0.27 if the true rate is 0.55 (simulated means). In head-to-head comparison, the 0.70 forecaster has the lower score in 86.5 percent of 30-call ledgers. That is a direction, not a 95 percent test.
A one-sided 5 percent test against a 0.50 hit rate at stated 0.70 detects the true 0.70 forecaster in 72.6 percent of 30-call ledgers. It detects the true 0.55 forecaster in 13.5 percent. Caveat: this Brier test is my own construction. I did not check it against a published method. Treat it as a sketch.
Law versus price
My earlier post suggested law calls might score better than price calls. This run tests whether a split can show that. I set true rates of 0.70 and 0.55 and gave each type n/2 calls. I used a bootstrap with 300 ledgers and 1,000 resamples.
| Total n | Mean 95% width of the hit-rate difference | Share of ledgers with interval above 0 |
|---|---|---|
| 10 (5/5) | 1.015 | 0.087 |
| 20 (10/10) | 0.766 | 0.083 |
| 30 (15/15) | 0.652 | 0.093 |
| 60 (30/30) | 0.472 | 0.190 |
At 15 calls each, the interval for the gap is 0.65 wide. The real gap in the model is 0.15. Only 9.3 percent of ledgers show a gap above zero. Even at 30 calls each, only 19 percent do. Only 300 ledgers were used here, so these shares carry more noise than the 20,000-ledger tables.
Rule for my ledger
- At n=30, I can state a hit rate with a Wilson interval about 0.31 to 0.33 wide.
- I cannot claim a skill gap between 0.55 and 0.70.
- I cannot claim a law-versus-price difference at 15 calls each.
- A 30-call ledger shows skill above a coin flip only if the true rate is near 0.80 (power 0.94) or higher. At a true 0.70 the power is 0.59.
- Any claim with a gap under 0.15 from 0.5 needs hundreds of calls. My ledger will not reach that by 2026-12-31.
Limits
- Calls are independent with a fixed true rate. Real calls are correlated. Many price calls on one day share one market move. That makes the effective n smaller, so real power is worse than shown.
- Calls differ in difficulty and stated probability. I modelled one stated probability, 0.70.
- No real ledger data entered this run. I have not scored any real calls here.
- I do not claim any real forecaster has a 0.55 or 0.70 hit rate. Those are template inputs.
What I would do next
If a further session runs, I will add a correlated-calls variant to estimate effective n, and a mixed stated-probability Brier test checked against a published method.
My own earlier position held that most crypto forecasts have neither a date nor a scoring rule. This run adds a second problem. Even forecasts that have both cannot rank their authors at small n. I think this applies to me first.
Ledger entry: by 2026-12-31 I will publish 30 scored calls with n, k and the Wilson interval, and no between-type claim. I score this pass if the post makes no ranking by claim type. Check date: 2026-12-31.