Real life · Regression to the mean
The Speed Camera That Cut Accidents by 70% Before It Was Switched On
The short version: if you pick places, people or teams because they just had an unusually bad year, they'll usually do better next year whatever you do. In our simulation, roads that averaged one accident a year and were picked after a year with three or more had about 3.3 accidents that year and 1.0 the next, with no change at all: a 70% 'improvement' from nothing. To know whether a speed camera, a new coach or a cure worked, compare with similar places picked the same way that didn't get it.
Mill Lane had three accidents last year, one of them involving the chairman's bicycle and a pothole. The parish council voted to install a speed camera. It cost $25,000 and was bolted to a lamppost in March.
This year, Mill Lane had one accident. The council issued a press release: "SPEED CAMERA CUTS ACCIDENTS BY 70%". Grandpa Frank, who walks past the camera every morning, wrote to the newspaper to point out that the camera wasn't connected to anything. The council replied that the camera had clearly worked, since there had been fewer accidents, and that its connection was therefore unnecessary.
Why bad years are followed by better ones
Accidents on a quiet road are mostly luck. Say Mill Lane, like hundreds of similar roads, has on average one accident a year. Some years it has none, some years two, and occasionally three or more, just by chance.
Now, which roads get speed cameras? The ones that just had a bad year. Councils naturally act where the numbers look worst. But a road picked for a bad year was mostly picked for its bad luck, and luck doesn't repeat on schedule. Next year it'll probably drift back towards its normal one. That drift is called regression to the mean.
The camera that didn't need to work
We simulated 200,000 roads with an average of one accident a year and picked the ones that had three or more in a given year, just as a council would.
- The picked roads averaged about 3.3 accidents in their bad year.
- The next year, with no camera, no change, nothing at all: about 1.0.
- That's a fall of about 70%, produced entirely by how the roads were chosen.
Any camera put on those roads would have "cut accidents by 70%". So would a lucky horseshoe, a strongly worded sign or a committee. This doesn't mean speed cameras don't work. Studies that account for regression to the mean generally find that they do reduce accidents, by less than the raw before-and-after numbers suggest. The point is that before-and-after alone can't tell you.
Accidents per road, roads picked after a bad year
The same trap everywhere
Regression to the mean turns up wherever something is picked because it was extreme. The football manager sacked after a run of defeats, whose replacement "turns things round". The employee sent on a course after a terrible quarter. The hangover cure taken at your worst moment. In every case, things were likely to improve anyway, and whatever happened in between gets the credit.
How to tell a fix from a fluke
1. Ask why this one was picked. If it was chosen because of an unusually bad (or good) result, expect it to move back towards normal on its own.
2. Look at more than one year before. Was last year typical or a spike? A spike will usually fall.
3. Compare with a control group. Find similar roads that had a bad year but didn't get a camera. If they improved just as much, the camera did nothing extra.
Where it costs you money
Investors often switch into last year's best fund and out of last year's worst. Part of an extreme year is usually luck, so the star tends to cool and the dog tends to recover. A strategy that "fixes" a bad year will often look clever simply because the bad year was unusual. Judge any change against what would have happened without it.
The moral
The council has voted to install a second camera on Station Road, which had four accidents last year. Grandpa Frank has offered to save the council $25,000 by standing on Station Road with a cardboard box painted grey. He predicts a 75% reduction. The council is considering his offer, but has asked whether the box would need to be connected.
Mill Lane and the council are fictional; the 3.3 and 1.0 figures come from a simulation of roads averaging one accident a year. Real speed camera evaluations that control for regression to the mean generally find a genuine, smaller effect. Educational content, not financial advice.