Statistics From Zero · Lesson 18 of 22
Simpson's Paradox: Better in Every Group, Worse Overall
In one sentence: A trend that shows up in every group can reverse when the groups are combined, because the groups are mixed in different proportions, so always ask what's inside an overall number.
The idea
Here's a statement that sounds impossible: A can be better than B in every single group, and worse than B overall. It isn't a trick or a rounding error. It happens in real data surprisingly often, and it has a name: Simpson's paradox.
The cause is always the same. The groups are mixed in different proportions for A and B, and the overall number blends them together without telling you.
The family quiz
The family holds a quiz night. Questions come in two piles, easy and hard, and you choose which pile to draw from. At the end, Uncle Gary has answered 72% of his questions correctly and Aunt Linda only 39%. Gary does a lap of the living room.
Linda asks to see the breakdown.
| Easy questions | Hard questions | Overall | |
|---|---|---|---|
| Linda | 9 of 10 (90%) | 30 of 90 (33%) | 39 of 100 (39%) |
| Gary | 70 of 90 (78%) | 2 of 10 (20%) | 72 of 100 (72%) |
Linda did better on easy questions (90% against 78%). Linda did better on hard questions (33% against 20%). Gary "won" overall, because he took 90 easy questions and Linda took 90 hard ones. The overall percentage isn't measuring quiz skill. It's mostly measuring which pile you chose.
The lap of the living room was cancelled.
What's going on
An overall rate is a weighted average of the group rates, and the weights are how many cases each person had in each group. Gary's overall number is mostly his easy-question rate (90 of his 100 questions). Linda's is mostly her hard-question rate. Comparing them compares easy questions with hard ones, not Gary with Linda.
The difficulty of the question is a confounding variable (Lesson 7): a hidden third thing that affects both who answered what and how well it went. Whenever the groups differ in mix, the overall comparison can point the wrong way.
Where it hides in real life
- Admissions: a famous university study from the 1970s found a lower overall admission rate for women, while most individual departments admitted women at similar or higher rates. Women had applied more to the departments that were hardest to get into.
- Rankings: a school, branch or team with a worse overall score may simply handle the hardest cases.
- Investments: one fund can beat another in every sector it holds and still trail overall, because it held more of the sectors that had a bad year. Always compare like with like: same markets, same period, same risk.
How to protect yourself
1. Ask what's inside the overall number. Are the groups mixed in the same proportions on both sides?
2. Look at the breakdown when it exists. If every group points one way and the total points the other, trust the groups, then ask why the mix differs.
3. But don't split forever. Cut data into enough groups and something will reverse by chance (Lesson 8's multiple testing). Split only by things that plausibly matter, like question difficulty.
Try it yourself
Two pizza branches compare how often deliveries are late. Branch A: 31 late out of 200 (15.5%). Branch B: 18 late out of 200 (9%). Head office plans to retrain Branch A. Then someone splits it by time of day:
Rush hour: A 30 late of 150, B 12 late of 50. Quiet hours: A 1 late of 50, B 6 late of 150.
Which branch is actually better at delivering on time?
Show the answer
Rush hour: A is late 20% of the time, B 24%. Quiet hours: A 2%, B 4%. Branch A is better in both. It looks worse overall only because three quarters of its deliveries were at rush hour, when everyone is late more often, while three quarters of B's were in quiet hours. Head office was about to retrain the better branch. Kevin, who manages Branch B, has asked that this analysis not be circulated.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Simpson's paradox | A trend in every group that reverses (or disappears) when the groups are combined |
| Aggregated data | Data combined into one overall figure |
| Stratified (disaggregated) data | Data split into groups (strata), such as easy and hard questions |
| Confounding variable (lurking variable) | A hidden factor that affects both the grouping and the outcome |
| Weighted average | An average where some groups count more because they're bigger |
| Stratification / controlling for a variable | Comparing within groups so the hidden factor can't distort the result |
Three things to remember
- Better in every group and worse overall is possible, whenever the groups are mixed differently.
- An overall rate is a weighted average: always ask what the weights are.
- Split by things that plausibly matter, not by everything until something flips.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.