Corporate life · Small samples
A Strong Correlation in a Team of Eight: What It's Worth
The short version: in a small group, strong correlations appear by chance all the time. With eight people, two completely unrelated answers correlate at 0.5 or more (either way) 21% of the time. Ask ten questions and you have 45 pairs to compare: in our simulation, at least one pair reached 0.7 in 92% of surveys, and the strongest pair was typically around 0.81, with nothing real going on. With 50 people, a 0.5 correlation between unrelated things almost never happens. A correlation's size means little without the sample size.
Kevin's team of eight filled in a ten-question engagement survey. HR has run the numbers and found a striking result: people who rate the coffee machine highly also rate their manager highly. The correlation is 0.81. HR has concluded that good coffee builds trust in management and has ordered a better machine.
Darren rated both of them 2 out of 10, because he was in a bad mood that morning. Kevin suspects this matters.
What a correlation of 0.81 usually means
A correlation measures how closely two things move together, from minus 1 to plus 1. With hundreds of people, 0.81 would be very strong evidence of a real link. With eight, one or two people who answered everything high or everything low can produce it on their own.
How often chance alone does it
We simulated pairs of completely unrelated answers, many times over, for groups of different sizes:
| People answering | Unrelated answers correlate at 0.5 or more (either way) |
|---|---|
| 8 | 21% |
| 20 | 2.5% |
| 50 | under 0.1% |
With eight people, one pair of unrelated questions in five correlates at 0.5 or more. The sample size is what tells you whether a correlation is surprising.
How often two unrelated answers correlate at 0.5 or more (either way)
Ten questions, 45 chances
The survey had ten questions, which makes 45 different pairs to compare. We simulated eight people answering ten questions with no connection between any of them:
- At least one pair correlated at 0.7 or more in 92% of surveys.
- On average, 9 of the 45 pairs reached 0.5 or more.
- The strongest pair was typically around 0.81.
So HR's headline result is exactly what an eight-person survey of unrelated questions would usually produce. Looking at 45 pairs and reporting the strongest is multiple testing, and in a small sample it finds something every time.
Statistics for noobs: size first, then strength
Before you get excited about a correlation, ask two things: how many people or data points is it based on, and how many correlations were looked at to find it? A rough rule: with fewer than about 30 cases, only very strong correlations stand out from chance, and with many pairs checked, even those can be luck. Better still, test the pattern again on a new group.
Where it costs you money
"This indicator correlated 0.8 with the market over the last eight quarters" is an eight-point correlation, probably one of many indicators tried. It's the same trap. Ask for many more data points, and for the result on a period that wasn't used to find it.
The moral
The new coffee machine arrived. In the next survey, coffee ratings rose and manager ratings fell, and the correlation was minus 0.6. HR has concluded that the new coffee is undermining management and is reviewing the contract. Kevin has pointed out that the strongest correlation in the new survey was between "I understand our strategy" and the number of letters in people's surnames. HR is reviewing that too.
What this model can't tell you
- Real answers aren't independent. In real surveys, people's moods colour all their answers, which creates genuine but uninteresting correlations on top of chance.
- Our model uses smooth scores. Real surveys use 1-to-5 or 1-to-10 scales, which makes small-sample correlations even jumpier.
- Small teams can still learn things. Talking to eight people is useful; computing correlations between their answers mostly isn't.
HR, the coffee machine and Darren's mood are an illustration. The figures come from our own simulations of unrelated, normally distributed answers: 400,000 pairs per group size, and 100,000 surveys of ten questions answered by eight people. Educational content, not HR or financial advice.