Statistics From Zero · Lesson 8 of 22
Is the Coin Rigged? Hypothesis Testing and the P-Value
In one sentence: To decide whether something is real or just luck, assume it's just luck and ask how surprising your result would be; the p-value is that surprise, written as a probability.
The idea
Uncle Gary flips a coin to decide who does the dishes. It's come up heads, Gary's call, an awful lot lately. Is the coin rigged, or is the family just unlucky?
You can never prove a coin is rigged by flipping it. Even a perfectly fair coin can land heads ten times in a row. What you can do is ask: if the coin were fair, how surprising would this result be? If the answer is "very surprising", you start to doubt the coin. That's all hypothesis testing is.
How it works, in four steps
- Step 1. Assume nothing interesting is going on. The coin is fair. The drug does nothing. The trading strategy is no better than random. This boring starting assumption is called the null hypothesis.
- Step 2. Collect the data. Flip the coin, run the trial, test the strategy.
- Step 3. Ask how surprising the data is if the boring assumption were true. That probability is the p-value.
- Step 4. Decide. If the p-value is small (traditionally below 5%), the data would be quite surprising under the boring assumption, so you reject it. The result is called statistically significant. If not, you don't have enough evidence to reject it.
It's the same logic as a courtroom. The defendant is presumed innocent (the null hypothesis). The prosecution has to produce evidence that would be very unlikely if they were innocent. "Not guilty" doesn't mean "proven innocent". It means the evidence wasn't strong enough.
The dishes coin goes on trial
Aunt Linda confiscates the coin and flips it 10 times. It lands heads 8 times.
If the coin were fair, how often would you get 8 or more heads out of 10? Counting all the possibilities, it's 56 out of 1,024, about 5.5%. That's the p-value.
Suspicious, but not quite under the traditional 5% line. One test of 10 flips can't convict the coin. Linda, who is thorough, flips it another 90 times. In 100 flips it lands heads 60 times.
The chance of 60 or more heads out of 100 from a fair coin is about 2.8%. Now the coin is officially under suspicion. Gary claims it's "just a lucky coin". Linda points out that this is exactly the problem.
Notice what happened: 60% heads over 100 flips is more convincing than 80% heads over 10. A big effect in a tiny sample is easy to get by luck. A smaller effect in a bigger sample is much harder (Lesson 6 again).
Ten flips of a fair coin: how often each number of heads turns up
What the p-value does NOT mean
The p-value is the most misunderstood number in science. Three classic mistakes:
- It is not the chance that the null hypothesis is true. A p-value of 2.8% doesn't mean "there's a 2.8% chance the coin is fair". It means "if the coin were fair, a result this extreme would happen 2.8% of the time". The difference sounds small. It isn't.
- "Not significant" doesn't mean "no effect". After 10 flips, the coin wasn't significantly unfair. That didn't prove it was fair. It meant 10 flips weren't enough to tell.
- "Significant" doesn't mean "big" or "important". With a huge sample, a coin that lands heads 50.1% of the time will eventually be "statistically significant". That's real, but nobody should care about it.
The trap: testing lots of things
A 5% threshold means that when nothing is going on, you'll still get a "significant" result 5% of the time, by luck. Test one thing and that's an acceptable risk. Test 20 things that all do nothing, and the chance that at least one of them looks significant is about 64%.
This is called multiple testing, and it's how a lot of nonsense gets published: try many versions, report the one that "worked". The fix is to adjust the threshold for the number of tests, or to check the winner again on fresh data it has never seen.
Why anyone outside class cares
Every trading idea we test on this site goes through exactly this. The null hypothesis is "this strategy is no better than buying a random stock on the same day". We measure how surprising the strategy's results would be if that were true, and because we usually test several versions of each idea, we correct for multiple testing. Most famous strategies don't survive the correction. Some looked significant before it, which is precisely why the correction exists.
Try it yourself
Aunt Linda claims she can tell expensive wine from cheap wine. Kevin sets up a test: 10 glasses, each one randomly filled with either expensive or cheap wine, and Linda doesn't see the bottles. She gets 9 out of 10 right.
What's the null hypothesis? Is the result statistically significant at 5%? And does it prove Linda can taste the difference?
Show the answer
Null hypothesis: Linda is guessing, so each answer is a 50/50 coin flip.
p-value: the chance of getting 9 or more right out of 10 by guessing is 11 out of 1,024, about 1.1%. That's below 5%, so the result is statistically significant. Guessing is a poor explanation.
Does it prove it? Not quite. It's strong evidence, but a 1.1% fluke still happens about once in 90 tests. If Kevin had tested the whole family and Linda was the one who did well, that's multiple testing again. The honest next step is to repeat the test with fresh glasses. If she does it again, Uncle Gary has to stop buying the cheap one and pouring it into the expensive bottle.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Null hypothesis (H0) | The boring assumption: no effect, no difference, just luck |
| Alternative hypothesis (H1) | The interesting claim you're testing: there is an effect |
| Test statistic | A number that summarises how far the data is from what the null hypothesis expects |
| p-value | If the null hypothesis were true, the chance of a result at least this extreme |
| Significance level (alpha, α) | The cut-off you pick in advance, usually 5% |
| Statistically significant | The p-value came out below the cut-off |
| Type I error (false positive) | Rejecting the null hypothesis when it was actually true: convicting an innocent coin |
| Type II error (false negative) | Failing to reject it when there really was an effect: letting a rigged coin go |
| Power | The chance a test spots a real effect. Bigger samples give more power |
| One-tailed / two-tailed test | Whether you only care about one direction (too many heads) or both (too many heads or too many tails) |
| Multiple comparisons | Running many tests at once, which makes flukes much more likely |
Three things to remember
- Hypothesis testing assumes it's just luck, then asks how surprising the data would be. The p-value is that surprise.
- A p-value is not the chance the null hypothesis is true. "Not significant" isn't proof of no effect, and "significant" isn't proof of a big one.
- Test enough things and something will look significant by luck. Always ask how many things were tried.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.