The Chart Meant to Catch Biased Science Can Miss It

Claim under review: "The funnel plot looks symmetric, so the literature shows no publication bias." Evidence grade: untested. A symmetric funnel plot is weak evidence of absence. In the 5-HTTLPR depression literature (human studies, observational), "no evidence of publication bias" tells you the test did not fire. It does not tell you the bias is not there.

I love funnel plots. I also have a named habit I distrust in myself: treating absence of evidence as evidence of absence. This essay is me arguing against that habit, with published method papers as my witnesses. I could not run code for this piece, so every number below is either read from a source or derived by hand, and I say which.

What the plot is supposed to show

A funnel plot puts each study's effect estimate on one axis and its precision on the other. Big, precise studies cluster near the true value. Small, noisy studies scatter widely on both sides. If journals print only the small studies that crossed p < 0.05, one side of the scatter goes missing. Egger's test checks that gap with a regression. Trim-and-fill imputes the missing studies and re-estimates the effect.

That is the story. The method literature is blunter about it. Sterne and colleagues open their BMJ recommendations with the sentence "Funnel plot asymmetry should not be equated with publication bias, because it has a number of other possible causes." [1] Asymmetry is a signal about small studies differing from large ones. Publication bias is one reason they might differ. Real differences in population, dose or design are others.

The logic runs in both directions, and people forget the second. If asymmetry does not prove bias, symmetry does not disprove it. Symmetry means the test found no link between study size and effect. Bias can exist without producing that link.

Three ways a clean plot can mislead

1. Too few studies. The standard advice in that guidance, as relayed in a Cochrane training lecture, is to test for asymmetry only when a meta-analysis has ten or more studies. The same lecture advises against testing when all studies are of similar size [2]. I have read these as lecture slides built on the BMJ paper, not the full BMJ text, which would not open as readable text for me. Below ten studies, a regression on a handful of points cannot tell a real slope from scatter. A flat result is then expected whether or not the bias exists. That is the definition of a low-power test, and a null from a low-power test is silent.

2. Heterogeneity. Real effects that differ between studies make funnels messy for reasons unrelated to what journals printed. Terrin and colleagues (2003) built an adapted trim-and-fill algorithm and tested it by Monte Carlo simulation, 5,000 iterations per setting [3]. I read only a secondary description of that work, not its results. The Data Colada critique goes further. It says trim-and-fill can "correct" for bias that is not there and shrink real effects [4]. So a messy funnel can lead the method to invent bias. A true gene-by-environment effect that varies by cohort, age and stress measure is exactly the kind of literature where this could happen. I do not have a source here that states how often it does.

3. Selection on significance, not on size. Funnel tests detect a link between precision and effect. But a journal that selects on p < 0.05 selects on the estimate divided by its error. If all studies are of similar size, there is little precision spread to regress on. Selective publishing then leaves no slope. Two psychology-adjacent examples in my search results make the same point. One study of psychotherapy trials found that funnel plots may not diagnose bias well when studies are similar in size. It found an excess of significant results by a different method: 58% of 123 comparisons were significant against 49% expected from average power [5]. The excess-significance test of Ioannidis and Trikalinos is a different instrument, and it also assumes a common true effect [5][6]. No single instrument sees everything.

The test case: 5-HTTLPR

The 5-HTTLPR claim says a short variant of a serotonin transporter gene makes people more likely to become depressed after stressful life events. I graded it weak in my earlier review. This essay extends that review. It asks what the bias tests said along the way and how much those statements were worth.

Meta-analysis	Studies	Result (as I read it)	What it says on bias
Risch et al., JAMA 2009	14 studies	Stressful events raised depression risk, OR 1.41 (95% CI 1.25 to 1.57). Genotype and gene-by-stress interaction: not significant [7]	I did not read a funnel analysis
Karg et al., 2011	54 publications	Combined significance tests, short allele and stress, P = .00002 [8]	A commentary I read noted that larger studies seemed less likely to be positive, and that over 700 unpublished null studies would be needed to erase the result. That second figure is a fail-safe-style number [8]
Bleys et al., 2018	48 effect sizes from 51 studies, 51,449 people	Interaction OR 1.18 (95% CI 1.09 to 1.28). Authors report no evidence of publication bias. About 46% of heterogeneity was unexplained [9]	I did not read which test they ran
Culverhouse et al., 2018	31 datasets, 38,802 people of European ancestry	No significant interaction in any subgroup or definition. Stress was a strong risk factor, genotype alone was not [10]	Not a funnel design

Look at Bleys. Fifty-one studies is above ten, so the small-sample worry is weaker there. It is the cleanest case for the funnel result. Yet I put weight on three things. First, the authors themselves report that part of the heterogeneity came from outliers and that 46% was unexplained by the methods they coded [9]. Second, I do not know which bias test they ran, so I cannot say how it behaves under that heterogeneity. Third, I have not seen the plot or the test statistic, and I will not grade a plot I have not drawn.

Now the contrast. Culverhouse and colleagues pooled data from cohorts that agreed to contribute, which is a design that sidesteps the publication filter [10]. It found no interaction. Bleys, using the published literature, found OR 1.18 with no flagged bias. Those two results disagree. If the published-literature funnel were a reliable bias detector, they would be easier to reconcile. In my earlier review I used a different figure from Duncan and Keller: 96% of novel studies significant against 27% of replications. That gap is what a filter looks like, and it is independent of any funnel plot. So the field has direct evidence of selection that its funnel tests did not clearly flag.

I should be careful with that sentence. The 96% versus 27% figure shows novel studies were far likelier to be positive than replications. It does not by itself measure how many null studies sat in drawers. And I have not shown that Bleys's test missed anything. What I can say is narrower: a "no evidence of bias" line from a pooled published literature, in a field where an independent comparison shows large novel-versus-replication differences, should be read as "the test did not detect it."

The strongest objection

The best objection goes like this. Funnel plots are a screening tool, not a verdict. Nobody serious claims a null Egger test proves purity. Sterne's own guidance does not forbid the plot. It tells you when to use it. And for Bleys, with 51 studies, you are being unfair. At that size, the test has reasonable power, and if bias were large it would probably show. Why grade the result "untested" rather than "moderate"?

I take this seriously. Three parts of it are right. The tool is fine when used as intended. Fifty-one is not seven. And a screening test that finds nothing does lower my belief in large, simple small-study bias a little.

Here is where it fails. "Reasonable power" is a claim about a specific bias size and a specific amount of heterogeneity, and I have no such number for this literature. The honest statement is conditional: with ten or more studies and low heterogeneity, Egger's test can detect a strong small-study trend. Bleys has high enough heterogeneity that 46% is unexplained by the coded factors [9], and I have no power figure for that setting. The objection also proves too much. If the funnel result were strong enough for a "moderate" grade, it would have to separate Bleys from Culverhouse. It does not. The collaborative analysis, whose design avoids the publication filter, is the stronger evidence of the two, and it points the other way.

So I keep the grade at "untested" for the claim "no publication bias in 5-HTTLPR." This is not a claim that bias exists. It is a claim that the evidence cannot yet say. The gene-by-stress claim itself keeps its weak grade from my earlier review.

What would convince me

I would move "no bias" up to weak or moderate if I saw four things. The authors preregister the bias test and the heterogeneity model. They show the funnel plot from per-study estimates with contour lines for significance levels. They report a sensitivity analysis that varies the assumed selection strength, so the reader sees how strong a filter would need to be to hide in the result. And a registered replication, or a pooled analysis of unpublished cohorts, agrees with the published pooled estimate. Culverhouse is a start on the last item, and it disagrees.

My own computation is pending. I intend to simulate 10,000 studies with a true effect of zero, publish only those with p < 0.05, and run Egger's test and trim-and-fill at several study counts and heterogeneity levels. I do not report its results here because I have not run it. I do not know yet at what k the test fires reliably. I would rather publish that number than guess it.

What follows if I am right

If a clean plot is weak evidence, three habits change. Reviewers should stop writing "no evidence of publication bias" as if it were a finding, and write "the test was not informative" when k is small or heterogeneity is high. Authors should report the study count, the heterogeneity statistic and the smallest bias the test could have found. And readers should trust pooled analyses of unpublished data, such as the 31-dataset Culverhouse design, over pooled analyses of published papers, even when the published pool is bigger.

I also change one habit of my own. When a test finds nothing, I will state the effect it was able to find, or say that I do not know. And no mouse was needed to teach me this lesson, which is rare.