Corporate life · Murphy's laws
The most dangerous equation: why the best and worst offices are both tiny
The short version: when you rank groups by a rate or an average, the smallest groups end up at both ends, because small samples swing more. The typical error of a percentage measured on n people is the square root of p(1 - p)/n: about 13 points for 5 people, 2 points for 200. In our model of 50 small and 50 large offices with exactly the same true rate of 10%, small offices took 98% of the top 10 places and 100% of the bottom 10. Before rewarding or punishing the extremes, check how big they are.
Head office has ranked all 100 branches by staff turnover. The ten best branches lost nobody this year, and all ten are tiny. A memo is drafted: "Small is beautiful. Proposal: split every large office into teams of eight."
Kevin scrolls to the bottom of the same ranking. The ten worst branches, where a third or half of the staff left, are also tiny.
The law
The statistician Howard Wainer called it "the most dangerous equation" in a 2007 article in American Scientist. Abraham de Moivre worked it out in the 18th century: the typical error of an average shrinks only with the square root of the number of things being averaged. Wainer's examples included US counties with the lowest and the highest rates of kidney cancer, both mostly small rural counties, and a costly push for small schools after the best-performing schools turned out to be small. So were many of the worst.
The model
We built 50 small offices (4 to 12 people) and 50 large ones (80 to 300), and gave every single person the same 10% chance of leaving during the year. No office is better or worse than any other. Then we ranked them, many times over:
| Group in the ranking | Share that are small offices |
|---|---|
| Top 10 (lowest turnover) | 98% |
| Bottom 10 (highest turnover) | 100% |
In a typical run, small offices ranged from 0% to 50% turnover while large ones stayed between 6% and 16%. An office of eight with a true rate of 10% has a 43% chance of losing nobody at all. That's the whole "small is beautiful" result: chance, and nothing else.
How much noise to expect
The standard error of a percentage measured on n people is the square root of p × (1 - p) ÷ n. With a true rate of 10%, it's about 13 percentage points for 5 people, 9 for 10, 4 for 50 and 2 for 200. A ranking that mixes those sizes is mostly a ranking of size.
Typical chance error in a measured turnover rate, by office size
Statistics for noobs: small samples swing more
This is the sample size effect at its most practical. Before reading anything into a top or bottom of a list, check whether the extremes are the small groups. If they are, compare each group with what chance alone would give at its size, or pool small groups until they're big enough to say something.
Where it costs you money
Fund rankings have the same problem: small, young funds dominate both the top and the bottom of the one-year tables. A small fund's brilliant first year says much less than it seems, and so does a terrible one.
The moral
Kevin sent the memo back with the bottom of the ranking attached. Head office withdrew the proposal to split the large offices and replaced it with a proposal to close the ten small offices at the bottom. Kevin pointed out that next year a different set of small offices would be at the bottom. Head office said that in that case they would close those too.
What this model can't tell you
- In our model the true rate is identical. In reality some offices really are better; the noise just makes it hard to see which.
- Small can be genuinely different. Size can change how people work; the point is that the ranking alone can't show it.
- Fixes exist. Funnel plots, several years of data or statistical shrinkage towards the average make fairer comparisons.
The branches, the memo and head office are an illustration. "The Most Dangerous Equation" is Howard Wainer's article in American Scientist (2007). The figures are our own simulation of 50 small and 50 large offices with a true turnover rate of 10%, repeated 20,000 times. Educational content, not business or financial advice.