How to Lie With Statistics (So You Can Catch It)

In one sentence: Most misleading statistics aren't false; they're true numbers chosen, framed or measured so that they point the wrong way, and a handful of simple questions catches almost all of them.

The idea

The most dangerous statistics aren't made up. Made-up numbers are easy to catch. The dangerous ones are perfectly true numbers that have been chosen, framed or measured so they lead you to the wrong conclusion. Nobody has to lie. They just have to pick.

You've already met every trick in this course. This lesson puts them in one place, gives each one a face, and ends with a checklist you can use on any headline, advert or investment pitch for the rest of your life.

Trick 1: pick the start date

This is cherry-picking: choosing the time period, the examples or the data that tell the story you want. Whenever you see a result, ask: who chose the start and end dates, and what happens if you move them? A strategy that "has returned 500% since March 2009" might simply have bought at the bottom of the worst crash in decades. Almost everything did well from there.

Trick 2: chop the chart

A bar chart shows two pubs' customer ratings: 99 and 100. If the vertical axis starts at 0, the bars look identical, which is honest. Start the axis at 98 instead and the second bar looks twice as tall as the first. The data hasn't changed. The picture has.

Always look at where the axis starts. If it doesn't start at zero, the differences are smaller than they look.

Trick 3: say "doubles" instead of "one in ten thousand"

"Eating this snack every day doubles your risk of a rare illness!" Terrifying. But if the risk goes from 1 in 10,000 to 2 in 10,000, then for every 10,000 people who eat it daily, one extra person gets ill. The relative risk doubled. The absolute risk went up by one in ten thousand.

Both numbers are true. Headlines and adverts choose whichever one sounds more dramatic. Always ask for the absolute number: out of how many?

Trick 4: use a tiny sample

"4 out of 5 people prefer our product." Out of how many? If it was literally five people, one person changing their mind turns 80% into 60% or 100% (Lesson 6). Small samples produce dramatic numbers easily, so if the sample size isn't stated, assume it's small.

Trick 5: show only the survivors

"Many of the most successful founders dropped out of university!" True. But so did enormous numbers of people who didn't become successful founders, and you never hear about them, because nobody writes profiles of them. This is survivorship bias: looking only at the winners, because the losers have disappeared from view. Investment funds that did badly quietly close down. The ones left look brilliant on average, partly because the bad ones are no longer counted.

Trick 6: try everything, report the winner

Test 20 things that do nothing, and one of them will probably look "statistically significant" by luck (Lesson 8). Report only that one and you have a headline. This is called p-hacking or data dredging, and it's a polite form of the lucky-socks problem from Lesson 7. Ask: how many other things did they try?

Trick 7: pick the convenient average

"The average customer saved $500!" Mean or median? One customer who saved $40,000 can make the mean look wonderful while most people saved almost nothing (Lesson 2).

Trick 8: leave out the comparison

"Our strategy made 80% over ten years." Compared to what? If simply holding an index fund made more over the same ten years, the strategy lost money compared with doing nothing at all. A result without a comparison isn't a result. That's why every test on this site compares with the S&P 500 and with a random stock bought on the same days.

The checklist

Eight questions. Between them they catch almost every misleading statistic you'll ever meet:

• Compared to what? The market, a control group, doing nothing?
• Over what period, and who chose it? What happens if you move the dates?
• Out of how many? Sample size, and absolute numbers rather than "doubles".
• Which average? Mean or median, and is one outlier doing all the work?
• Who's missing? The failures, the people who didn't reply, the funds that closed.
• How many things were tried? Is this the one lucky winner out of many?
• Could something else explain it? A hidden third thing, or cause and effect the other way round?
• Where does the chart's axis start?

Try it yourself

The words your lecturer uses

Three things to remember

• Misleading statistics are usually true numbers, chosen and framed to point the wrong way.
• The two most powerful questions are "compared to what?" and "out of how many?"
• Be most suspicious of the numbers you'd most like to believe.

You finished part 1

That's the ten core lessons. If you want to go further, part 2 continues with Lesson 11: conditional probability and Bayes, then regression, expected value and the most common tests (t-test, chi-square). You now know more about how statistics actually works than most people who quote them on television. If a lecturer, a headline or Uncle Gary throws a number at you, you have the questions to ask. Go and ask them.

And if you'd like to see these ideas applied to real money, our tests of popular trading ideas use every one of them: control groups, sample sizes, costs, multiple-testing corrections and a comparison with the S&P 500.

Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.