Statistics From Zero · Lesson 23 of 25
Regression to the Mean: Why Extreme Results Don't Last
In one sentence: When a result is partly skill and partly luck, an extreme result usually contains extreme luck, so the next one tends to land closer to the average, with no cause needed.
The idea
Most results are a mix of two things: something stable (skill, quality, how dangerous a road is) and something that changes every time (luck, weather, a bad night's sleep). When a result is extreme, it's usually because both pointed the same way: decent skill and unusual luck. Next time the skill is still there, but the luck is redrawn. So the next result tends to be less extreme.
That's regression to the mean. It isn't a force or a law of nature that pulls things back. It's just what happens when you select something because it was extreme and then look again.
The coach's discovery
Aunt Linda coaches the kids' football team on Saturdays. Uncle Gary helps, in the sense that he stands next to her with a flask of coffee and opinions.
After a terrible match, Linda shouts at the team. The next week they play better. After a brilliant match, she praises them warmly. The next week they play worse.
Gary has drawn the conclusion. "Shouting works. Praise makes them soft." He has started shouting at the hedge, which has not improved.
Gary's evidence is real: after shouting, results did improve, and after praise they did drop. But it would have happened with no shouting and no praise at all. A terrible match is usually a mix of an ordinary team and bad luck. The next match draws fresh luck, so it's probably better. A brilliant match is usually an ordinary team plus good luck, so the next one is probably worse. The coach's words arrive exactly at the moments when the results were about to move anyway, and they get the credit or the blame.
A model you can check
Here's a simple model, with numbers we simulated for a million students. Each student has a skill level (average 60 points, standard deviation 10), and every test adds luck on top (standard deviation 10). They take two tests. Nothing changes between them: no extra teaching, no shouting.
| Group (picked by test 1) | Test 1 average | Test 2 average |
|---|---|---|
| Top 10% on test 1 | 84.8 | 72.4 |
| Everyone | 60.0 | 60.0 |
| Bottom 10% on test 1 | 35.2 | 47.6 |
The top group dropped by more than 12 points and the bottom group rose by the same amount, with no change at all. The top group is still above average (they really are more skilled on average), just not as far above as their lucky first test suggested. And the whole class is exactly where it was: nobody got worse overall. Only the groups picked for being extreme moved back.
How far back? The one-line formula
If the correlation between the first and second result is r (Lesson 7), then on average:
expected next result = average + r × (this result minus the average)
In the model, the two tests correlate at r = 0.5, so the top group's expected second score is 60 + 0.5 × (84.8 - 60) = 72.4. Exactly what the simulation shows. If results were pure skill (r = 1), there would be no regression at all. If they were pure luck (r = 0), the next result would be the plain average, whatever happened last time. Most real things sit in between. This is also where the word "regression" in linear regression (Lesson 12) comes from.
Where it fools people
- Speed cameras and blackspots. Cameras tend to go where there were the most accidents last year. Those spots were partly dangerous and partly unlucky, so accidents would fall the next year anyway. In a model of 50 junctions that are all equally dangerous (5 accidents a year on average), the worst five in one year average about 9.1 accidents, and the next year about 5.0, with nothing done at all. Any honest evaluation has to compare with similar junctions that got no camera.
- The best performer of last year. The employee of the month, the top salesperson, last year's best fund: picked because they were extreme, so they often look more ordinary next time. That's not necessarily a curse or laziness.
- Treatments that start when things are worst. People try remedies for a bad back, a slump or a losing streak at the lowest point, which is exactly when things were likely to improve anyway. That's why serious studies need a control group (Lesson 25).
Try it yourself
A town installs speed cameras at the 5 junctions that had the most accidents last year: on average 12 each. The next year those junctions average 7. The council announces that the cameras cut accidents by 42%.
1. Why might the 42% be too generous? 2. What extra information would let you judge the cameras fairly?
Show the answer
1. The junctions were chosen because last year was extreme. Part of that year was bad luck, which wouldn't repeat. So some of the drop from 12 to 7 would have happened with no cameras at all. The 42% mixes the camera effect with regression to the mean.
2. A comparison: junctions that were just as bad last year but didn't get cameras (or, better, a random choice of which bad junctions get them). Also several years of history before the cameras, so you know each junction's normal level rather than its worst year. If the camera junctions fell more than similar junctions without cameras, that extra drop is the cameras' effect. It may still be substantial. The point is to measure it, not assume it.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Regression to the mean (regression toward the mean) | Extreme results tend to be followed by less extreme ones when luck is part of the result |
| True score and measurement error | The stable part (skill) and the changing part (luck) of a result |
| Selection on the outcome | Choosing cases because their result was extreme, which sets up regression |
| Test-retest correlation (reliability) | How strongly two measurements of the same thing agree; the lower it is, the stronger the regression |
| Regression fallacy | Crediting a cause (shouting, a camera, a remedy) for movement that regression explains |
| Control group | A comparable group without the intervention, the standard way to separate the effect from regression |
Three things to remember
- Extreme results usually contain extreme luck, and luck doesn't repeat on request.
- Expected next result = average + r × (this result minus the average). The weaker the link, the bigger the pull back.
- Whenever something is chosen because it was the best or the worst, compare it with a similar group that got no treatment before crediting anything.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.