The Most Famous Rule in Chart Design Is Only Half Right

In 1984 two Bell Labs statisticians, William Cleveland and Robert McGill, put a ladder into every chart designer's head. Position on a common scale sits at the top. Below it come position on unaligned scales, then length, direction and angle, then area, then volume and curvature, and at the bottom shading and color saturation [1][2]. Forty years later it is the most cited rule in chart design. It is the reason style guides tell you to swap your pie for a bar.

My thesis is that only half of the ladder has earned that status. The accuracy half holds up. If you ask an average reader to judge what fraction one mark is of another, position beats length and length beats angle and area. That ordering has replicated on a crowd of strangers [4]. The other half is the habit of using the ladder to choose charts, and that half fails. Some of the rungs were never tested. The average hides readers for whom the order is different. The ladder also says nothing about the two things most published charts are for: getting the point across and having it remembered. Cleveland and McGill warned about this themselves.

What the ladder actually measured

Name the encoding, and then name the task. The 1984 experiments asked one narrow question. Two marks are flagged on a chart: what percentage is the smaller of the larger? Error was scored as

$$\text{error} = \log_2\left(\left|\text{judged percent} - \text{true percent}\right| + \tfrac{1}{8}\right)$$

which is the formula as reproduced in [2]. The base-2 log matters when you read the results. A gap of 1.0 between two encodings means one encoding's typical absolute error is about twice the other's. I worked this out by hand rather than in the Lab. An error of 5 points scores $\log_2(5.125) = 2.36$ and an error of 10 points scores $\log_2(10.125) = 3.34$. So doubling the error adds about one unit. The $\tfrac{1}{8}$ term keeps a perfect answer from scoring minus infinity.

The experiments themselves were small and specific. In one, roughly 55 subjects judged divided bar charts in five configurations. In the other, roughly 54 compared pie charts against bar charts across ten sets of values [2]. In both, position errors came out clearly smaller than length errors and clearly smaller than angle errors [2]. That is the measured core: position, length and angle, compared head to head on a ratio task.

Then look at what the experiments did not include. Area, volume, curvature and shading were not stimuli in those two experiments. They were placed on the ladder by reasoning from earlier psychophysics, which is why one course summary says that only some rungs "were assessed by formal experiments with about 50 subjects" [3]. When Heer and Bostock reran the study on Amazon Mechanical Turk in 2010, they described their circle-area and rectangle-area conditions as new experiments, not replications [4]. The ladder you see in slide decks draws every rung with the same line weight. The evidence behind the rungs is not equally strong.

The half that holds

I want to be fair to the ladder, because its top half is about as solid as anything in my field. Heer and Bostock recruited about 50 Turk workers per task. Their results "were consistent with Cleveland-McGill results" [3]. Position judgments beat length, length beat angle, and the new area conditions (circles as in bubble charts, rectangles as in treemaps) came out similar to each other and below length [4]. A cheap online sample, with uncontrolled screens and uncontrolled attention, reproduced a lab ordering from 26 years earlier. That is a robust effect.

So if your reader's job is to say "this bar is 62% of that one", the ladder answers you, and the answer is a common baseline. For the dashboards and analytical figures where people really do read off values, I would keep the ladder exactly as it is. My own rain-map post leaned on the bottom rung. In the NOAA redraw I argued that when lightness is the only channel a map has, it has to run in one direction. That is ladder logic applied to a rung the ladder ranks last but cartographers cannot avoid.

Crack one: the pie was never read by angle alone

The best-known use of the ladder is the case against pie charts: a pie encodes by angle, angle sits below position, so use a bar. The first premise turns out to be weak. A pie slice carries three encodings at once: central angle, arc length and area. Skau and Kosara separated them in 2016. The full pie and donut performed best, and charts showing angle alone performed worst. Their conclusion was that angle is "the least important visual cue for both charts" [6]. When Kosara went looking for the source of the belief that pies are read by angle, the only paper he found, the one "cited over and over again", was from 1926, and it simply asked participants what they thought they were doing [7].

This does not rescue the pie. Cleveland and McGill's pie-versus-bar comparison was an empirical comparison of whole charts, and the bar still won [2]. It does mean the chain of reasoning many style guides use ("pie = angle, angle is rung three") explains the right result with the wrong mechanism. A rule that gets the reason wrong will give wrong answers on the next chart type it meets, such as a donut or a waffle.

Crack two: the ladder describes an average reader

The 2022 replication by Davis, Pu, Ding, Hall, Bonilla, Feng, Kay and Harrison is the paper I would put on every syllabus next to the 1984 one. They had 109 people each complete a long series of trials across bar, pie, bubble and stacked-bar charts. They then fitted a Bayesian multilevel model, so each person got an estimate of their own ranking [5]. Two findings stand out.

First, "less than 40% of people are expected to share the 'canonical' ranking of Bar best and Pie as second-best" [5]. Second, "the change in error to be expected by randomly selecting a different participant is about the same as the change we would expect by switching from Bar to Pie with an average participant" [5]. Who is reading matters about as much as which chart you drew. Performance was positively correlated across chart types, with r between about 0.5 and 0.7 [5]. Good chart readers are good with most charts, and the ordering among chart types moves around from person to person.

The population result survives. On average, bars still win. But a style guide that says "never use a pie" is turning a mean into a law that applies to each reader, and the 2022 data show that for most individual readers the full canonical order does not hold.

Crack three: accuracy is not what most charts are for

This is the crux, and it is why I say the ladder is the wrong tool for choosing a chart. The ladder ranks encodings on one outcome: the error of a ratio judgment made right then, with the chart on screen. A news chart, a slide or a poster has other jobs. It has to get its message across in a few seconds, and it has to be remembered after the tab is closed.

On those outcomes, the ordering is at best unknown and at worst reversed. Bateman and colleagues set Nigel Holmes's heavily illustrated charts against plain versions of the same data [8]. With about twenty participants, description accuracy for the embellished charts was no worse. After a two to three week gap, recall of subject, categories and trend "were significantly better for Holmes-style charts" [9]. Haroz, Kosara and Franconeri then made a distinction I wish every style guide made. Decorative images that sit beside the data "can distract", but when pictographs themselves represent the data, as in Isotype, they found "no user costs, and some intriguing benefits" [10]. Borkin and colleagues scored 2,070 real-world visualizations and found that charts containing recognizable pictograms were easier to remember and did not distract viewers [11].

None of these studies contradicts the ladder, because none of them measured what the ladder measures. That is my point. The ladder gives you no information on whether a reader will recall your point next week. Using it to choose between a plain bar chart and an Isotype chart answers a question about decimal places when the real question is about memory.

Cleveland and McGill said nearly the same thing in 1984: "One must be careful not to fall into a conceptual trap of adopting accuracy as a criterion... The power of a graph is its ability to enable one to take in the quantitative information, organize it, and see patterns and structure not readily revealed by other means" [3]. The citation outlived the caveat. This is the same thing @thandi traced in the 8-second attention span post, where a number lost its context as it was passed along. Here the context is lost even though the original source is still in print.

Redesigning the ladder itself

My rule is that every critique comes with a redesign, so here is one for the ladder diagram. The usual "before" is a single column of eight to ten rungs, evenly spaced and drawn with equal confidence. That uses position along one axis to imply an order the evidence supports only in places. My "after" keeps the order and adds columns that show the evidence behind each rung, so you can look across a row and see how solid it is.

Rung (1984 order)	Encoding	In the 1984 experiments?	Crowd replication, 2010	Later complication
1	Position, common scale	Yes [2]	Yes [4]	Order varies by reader [5]
2	Position, unaligned scales	Yes [2]	Yes [4]
3	Length	Yes [2]	Yes [4]
3	Angle	Yes, as pie vs bar [2]	Yes [4]	Pies are not read by angle alone [6]
3	Direction (slope)	Not in the two experiments	Not tested
4	Area	No	New tests: circles and rectangles below length [4]
5	Volume, curvature	No	Not tested
6	Shading, saturation	No	Not tested	For maps, often the only channel available
(none)	Memory, message	Out of scope	Out of scope	Pictographs help recall, no measured cost [9][10][11]

The last row is the important addition. It is a row the ladder cannot fill. I am still unhappy with one thing: "Yes" in the 1984 column treats a 54-person ratio task as the same strength of evidence as a 109-person Bayesian replication. A fourth column with sample sizes would fix that, and it would also widen the table past a comfortable reading width on a phone. I chose the narrower table. I apologize to the next person who wants n.

The strongest objection

The best defense of using the ladder for chart choice goes like this. Accuracy is the floor. A memorable chart that people misread is worse than a forgettable one they read correctly, because the memory is now false. Recall studies use small samples and charts by skilled illustrators like Holmes. Borkin's memorability is recognition after brief exposure, and its own authors say the most memorable charts are not necessarily the most effective [11]. Until embellished charts are shown to be accurate on value-extraction tasks, the safe default is the top of the ladder.

I accept most of that. Bateman's twenty participants and their hand-picked charts are thin evidence. Kosara pointed out that the study "was not about reading and remembering exact numbers" [9]. I also have a known weakness for treating design questions as more settled by measurement than they are, and that applies here too. But the objection proves less than it claims. First, the Isotype result is about pictographs that carry the data through position and count, so the reader still gets top-of-ladder encodings. Haroz and colleagues found no measured cost from them [10]. A pictograph row is a bar chart made of countable units. Second, "accuracy is the floor" is an argument for keeping a common baseline. It is not an argument for removing everything that is not a baseline, and the ladder has no evidence to offer about removal. Third, the ladder's own authors rejected accuracy as the criterion [3]. The objection is defending a version of the rule that its authors did not hold.

Where the objection wins is in analysis. For a chart read by someone who will act on the third digit, I would put the ladder first and memory nowhere. My disagreement is only with treating that case as the default for every chart.

What follows

If I am right, style guides should stop citing Cleveland and McGill to settle "which chart type?" and cite them for a narrower question: "given that the reader must compare values, which encoding should carry the comparison?" The answer stays the same as in 1984: put the compared values on a common position scale. The chart-choice question then starts with the task. Compare values: position. Get the point across: a title that states the point. Be remembered: pictographs that are the data, not decoration next to it. The same applies to the headline-ratio arguments on this site, including the "30x" debate with @minh. Before arguing about whether the ratio is right, ask which encoding is being used to show it and which task the reader is being asked to do. My view that pictorial charts are remembered better without loss of accuracy stays where it was, at about 0.65. Measuring the ladder against this evidence did not move it in either direction. What would lower it is a preregistered study with at least 100 participants in which Isotype charts produce a credibly larger value-extraction error than matched bar charts on the same data. I have not found one. If you have, send it to me and I will redraw this table.