Are Employee Surveys Really Anonymous? The One That Needed Your Login
The short version: a survey result means nothing without the response rate. 47 happy people out of 50 replies is 94%, but in a company of 200 the true share could be anywhere from 23.5% to 98.5%. And if the 'anonymous' form can work out who you are, it doesn't measure happiness. It measures caution.
The email said: "Your voice matters! Our annual engagement survey is 100% anonymous." To open it, Kevin had to log in with his company account.
The first questions were: department, team, age band, years with the company. Kevin's team has five people. Only one of them is over fifty, and only one joined last year. Kevin is both.
Kevin rated everything "very satisfied". So, it turns out, did almost everyone else who answered.
94% happy. Of whom?
HR announced the result at the all-hands meeting: 94% of employees are satisfied. The small print, which nobody was shown, said 50 of the company's 200 employees had responded. 47 of them were satisfied. 47 out of 50 is indeed 94%.
But the question was about 200 people, and we've heard from 50. The other 150 said nothing. Here's what the true figure could be, depending on what the silent ones think:
| If the 150 who didn't answer are… | Satisfied, out of 200 | True share |
|---|---|---|
| all unhappy | 47 | 23.5% |
| exactly like the people who answered | 188 | 94% |
| all happy | 197 | 98.5% |
So the honest result of this survey is: somewhere between 23.5% and 98.5%. That isn't a finding. That's a weather forecast for the whole year. The 94% only holds if the people who stayed silent are just like the people who answered, and there is no reason to believe that. People who are busy, disengaged or job-hunting are exactly the ones who don't fill in engagement surveys.
200 employees, one "anonymous" survey
What the survey can really say
The anonymity that wasn't
Now the login. A company of 200 people has, say, 40 teams of five. Add four age bands and three bands for years of service, and you get 40 × 4 × 3 = 480 possible combinations for 200 people. Most combinations contain nobody. Many contain exactly one person. Answer the first four questions honestly and you have, in effect, signed the form.
The staff worked this out faster than HR did. A survey that can identify you doesn't measure how happy people are. It measures how careful they are. Statisticians call the first problem non-response bias: the people who answer are different from the people who don't. The second one doesn't need a name. Everyone who has ever worked in an office already knows it.
The process
The three people who said they were dissatisfied were invited to a "listening session" to discuss their concerns. HR has asked us to stress that this was entirely voluntary and entirely unrelated to the anonymous survey.
Next year, 30 people responded. All 30 were satisfied. Engagement has reached 100%, the highest in the company's history. Staff turnover has also reached its highest level in the company's history. These two facts are reported by different departments and have never been shown on the same slide.
How to read any survey result
1. Find the response rate. If it isn't stated, assume it's embarrassing. Below about half, the result tells you about the people who answered, not about everyone.
2. Work out the range. Assume all the silent people said no, then assume they all said yes. If the range is wide, the headline number is decoration.
3. Compare words with behaviour. What people say in a survey that knows who they are is weaker evidence than what they do: resignations, sick days, internal transfers. Behaviour is harder to fake and doesn't need a login.
Where it costs you money
Fund performance tables have the same silence problem. Funds that did badly often close or stop reporting, so the averages are built from the ones that are still around to answer. That's survivorship bias, and it's why every test on this site says which companies or funds might be missing.
The moral
A survey only hears the people who speak, and people only speak when it's safe. The company has now concluded that its culture is excellent, because the survey says so, because everyone who disagrees has stopped answering. The circle is complete. It is also anonymous.
The company, the survey and its numbers are an illustration, not data from a real organisation. Educational content, not financial advice.