Statistics From Zero · Lesson 15 of 22
Type I and Type II Errors: Why Small Tests Miss Things
In one sentence: Every test can be wrong in two ways, raising a false alarm or missing something real, and a small sample makes the second mistake far more often than most people realise.
The idea
In Lesson 8 you learned to test an idea by assuming it's false (the null hypothesis) and seeing whether the data is surprising enough to reject that. But the test can still get it wrong, in exactly two ways:
- Type I error, a false alarm: you declare an effect when there isn't one.
- Type II error, a miss: there is a real effect, and your test fails to spot it.
Statistics courses spend a lot of time on the first and very little on the second. In real life, the second is often the bigger problem.
The courtroom
A trial works exactly like a statistical test. The defendant is presumed innocent: that's the null hypothesis. The prosecution needs evidence "beyond reasonable doubt" to reject it.
| Really innocent | Really guilty | |
|---|---|---|
| Convicted | Type I error: an innocent person convicted | Correct |
| Acquitted | Correct | Type II error: a guilty person goes free |
Raise the bar of evidence and you convict fewer innocent people but let more guilty ones go. Lower it and the reverse. You can't push both errors to zero at once with the same evidence. The only way to reduce both is more evidence.
The 5% significance level is the courtroom's bar: you accept a 5% chance of a false alarm. The chance of a miss has its own name, and its opposite is the most important idea in this lesson: statistical power, the chance that your test catches an effect that really exists.
Kevin's lucky coin
Cousin Kevin has a coin he uses to decide who pays for drinks. Uncle Gary has paid suspiciously often. Gary suspects the coin is rigged. Let's say it really is: it lands heads 60% of the time instead of 50%.
Gary flips it 20 times as a test.
To call the coin rigged at the usual 5% level, Gary would need 15 or more heads out of 20. A coin that lands heads 60% of the time gets there only about 13% of the time. So 87 times out of 100, Gary's test says "no evidence the coin is rigged", and the coin is rigged. Kevin is delighted.
Here's how the chance of catching the coin grows with the number of flips:
| Flips | Heads needed to call it rigged | Chance of catching the 60% coin (power) |
|---|---|---|
| 20 | 15 or more | 13% |
| 100 | 59 or more | 62% |
| 400 | 217 or more | 99% |
The coin is exactly as rigged in every row. Only the amount of evidence changes. With 20 flips, the test is almost blind; with 400 it almost never misses. A common target in planning studies is 80% power, and for a 60% coin that takes something over 150 flips.
Why this matters for every "no effect found"
When a small study finds "no significant effect", there are two possibilities: there's no effect, or the study was too small to see it. Without knowing the power, you can't tell which. That's why "absence of evidence is not evidence of absence" is one of the most useful sentences in statistics.
It's also why we run our trading tests on thousands or millions of trades rather than a handful: with that much data, the tests have enough power to see even small edges. When an idea still shows nothing on hundreds of thousands of trades, "too small to see" is no longer a good excuse.
And the other side: a test with lots of power will flag even tiny effects as significant. With a billion coin flips you could prove a coin lands heads 50.01% of the time. Significant, and useless. Always ask how big the effect is, not just whether it's there.
Try it yourself
Uncle Gary tested his lucky socks over 10 pub quizzes and found "no significant effect". He concludes the socks don't work and throws them away. Aunt Linda says the test proves nothing either way.
Who's right, and what kind of error might Gary be making?
Show the answer
Linda, technically. Ten quizzes is a tiny sample, so the test had very low power: even if the socks helped a little, it would probably have missed it. If they do work, Gary is making a Type II error, a miss. "Not significant" with low power means "we couldn't tell".
That said, Gary's conclusion is still the sensible bet. There's no plausible reason socks would change quiz scores, so the prior (Lesson 11) is very low. Low power explains why the test can't rule it out; it doesn't make the socks more likely to work. Linda has quietly rescued them from the bin anyway, for the wash.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Type I error (false positive) | Rejecting the null hypothesis when it's actually true: a false alarm |
| Type II error (false negative) | Failing to reject the null hypothesis when it's actually false: a miss |
| Significance level, alpha (α) | The chance of a Type I error you accept, usually 5% |
| Beta (β) | The chance of a Type II error |
| Statistical power (1 - β) | The chance of catching an effect that really exists |
| Effect size | How big the real effect is (60% heads vs 50%); bigger effects are easier to catch |
| Power analysis | Working out, before a study, how much data you need to have a good chance of catching the effect |
Three things to remember
- A test can raise a false alarm (Type I) or miss something real (Type II). Only more evidence reduces both.
- Small samples have low power: "no effect found" often means "too small to tell".
- Huge samples find tiny effects significant: always ask how big, not just whether.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.