Statistics From Zero · Lesson 22 of 22
Statistically Significant Doesn't Mean Important: Effect Size
In one sentence: Statistically significant means a result is probably not luck, not that it's big or worth anything; to know whether it matters, look at the effect size, in real units, with its range.
The idea
In everyday English, "significant" means big or important. In statistics it means something much narrower: probably not just luck (Lesson 8). A result can be statistically significant and completely unimportant. And an important effect can fail to be statistically significant. Mixing the two meanings up is one of the most common mistakes in news headlines, company reports and, occasionally, exam answers.
The cure is a second number: the effect size, meaning how big the difference actually is.
The green button
Cousin Kevin has a job at an online shop. His team tested a green "Buy" button against the old blue one, with a million visitors seeing each. The blue button sold to 10.0% of visitors. The green one sold to 10.1%.
The test is statistically significant: p = 0.02. Kevin's manager has called it "a significant breakthrough", ordered every button on the site repainted and booked a meeting about green.
Kevin did the other calculation. The green button sells to one extra customer per thousand visitors.
Both numbers are correct. The p-value says the gap is probably real. The effect size, 0.1 percentage points, says how much it's worth. Whether that's worth a redesign depends on how many visitors the shop gets and what repainting costs, which is a business question, not a statistical one. What it certainly isn't is a breakthrough.
Why huge samples make everything significant
The p-value depends on two things: how big the effect is and how much data you have. With enough data, the standard error (Lesson 6) shrinks so far that even a tiny difference stands out from the noise. Here's the same effect at two sample sizes.
Two classes take the same exam. Class A averages 72, class B averages 70, and in both the standard deviation is 10 points.
| Students per class | Gap | p-value | 95% range for the gap |
|---|---|---|---|
| 20 | 2 points | 0.53 | from 4.4 points worse to 8.4 better |
| 2,000 | 2 points | less than 0.000001 | 1.4 to 2.6 points better |
The gap is the same 2 points both times. The p-value changed from "nothing to see" to "overwhelming" only because the classes got bigger. So a tiny p-value doesn't mean a big effect, and a large p-value doesn't mean no effect. It mostly tells you whether you had enough data to see the effect you've got.
Measuring the effect
- In real units first. "2 exam points", "1 extra sale per 1,000 visitors", "3 minutes faster". These are the numbers people can actually judge. For percentages, say whether you mean percentage points or a relative change (10.0% to 10.1% is 0.1 percentage points, or a 1% relative increase, which sounds much bigger).
- As a standardised size when units are hard to compare. The most common is Cohen's d: the gap divided by the standard deviation. For the exam, 2 ÷ 10 = 0.2. A rough, widely quoted guide calls 0.2 small, 0.5 medium and 0.8 large. It's a starting point, not a law: a "small" effect can matter a lot when millions of people are affected or money is at stake every day.
- With its range. A confidence interval (Lesson 9) shows the effect size and the uncertainty together. "1.4 to 2.6 points" says: definitely positive, definitely small. "4.4 points worse to 8.4 better" says: we can't tell yet. That's more useful than any p-value on its own.
The opposite mistake
Small studies can miss effects that matter. In Lesson 14, one route to work was 3 minutes faster on average, but with six trips each the result wasn't significant (p = 0.07). Three minutes every working day is about 12 hours a year. "Not significant" there meant "not proven with six trips", not "doesn't matter". The honest summary: a possibly useful effect, too little data, keep measuring.
Why this matters for trading results
This is exactly why every result on this site comes with costs and comparisons. As an illustration (a made-up example, not one of our tests): a strategy that beats the market by 0.02% per trade over a hundred thousand trades can be statistically significant. If each round trip costs 0.1% in fees and spreads, the strategy still loses money. Significant, real and useless. And the reverse applies to our own results: when we find no edge, that means "no edge we could detect with this data", not "proof that no edge exists".
Try it yourself
Uncle Gary reads that a study of 100,000 drivers found a brand of tyres gives significantly better fuel economy: 30.3 miles per gallon instead of 30.0 (p = 0.001). The tyres cost $200 more per set and last four years. Gary drives 12,000 miles a year, and fuel costs $3.50 a gallon.
1. Is the effect probably real? 2. How much would Gary save over the life of the tyres? 3. Should "significantly better" go on the box?
Show the answer
1. Probably yes. With 100,000 drivers and p = 0.001, luck is an unlikely explanation (assuming the study was done properly).
2. At 30.0 mpg he uses 12,000 ÷ 30 = 400 gallons a year. At 30.3 mpg, about 396.04. That's 3.96 gallons, or about $13.86 a year, so roughly $55 over four years, against $200 extra up front. Real, significant and not worth it on fuel alone.
3. It will, because it's technically true. The honest version is "about 1% better fuel economy", which tells you the effect size and lets you do the sum yourself. Gary has decided the tyres are significantly too expensive, which is the correct use of the word.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Statistical significance | The result is unlikely to be pure luck (p below a chosen level, often 0.05) |
| Practical (clinical, economic) significance | The effect is big enough to matter in the real world |
| Effect size | How big the difference or relationship is, regardless of sample size |
| Cohen's d | The gap between two averages divided by the standard deviation |
| Absolute vs relative difference | 0.1 percentage points vs "1% more"; the same change, sounding very different |
| Minimum important difference | The smallest effect worth acting on, ideally decided before the study |
| Overpowered study | One so large that trivial differences become significant |
Three things to remember
- Significant means "probably not luck", not "big" or "important".
- With enough data everything is significant; with too little, nothing is. The effect size doesn't care about sample size.
- Always ask: how big, in real units, with what range, and is that worth anything after costs?
Part 3 so far
Lessons 20 to 22 add three tools you'll meet in almost every course and every research report: the binomial distribution for counting successes, ANOVA for comparing more than two groups at once, and effect size for deciding whether a significant result actually matters.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.