Statistics From Zero · Lesson 6 of 22
Why Polls Only Ask About 1,000 People
In one sentence: A well-stirred sample of about 1,000 people pins down a whole country to within about 3 points, because what matters is the size of the spoon, not the size of the pot.
The idea
Every time a poll says "1,000 people were asked", somebody in the comments says: "There are 330 million people in this country. How can 1,000 tell you anything?"
Go back to the pot of chilli from Lesson 1. If the pot is well stirred, one spoonful tells you how salty it is. And here's the surprise: it doesn't matter whether the pot is a saucepan or a swimming pool. A properly stirred spoonful from a swimming pool of chilli is exactly as informative as one from a saucepan. What matters is the size of the spoon, not the size of the pot.
That's the core of this lesson: how accurate a sample is depends on how big the sample is, not on how big the population is. (As long as the sample is properly random. Hold that thought.)
Uncle Gary's chilli survey
After the barbecue disaster, Uncle Gary wants proof that his chilli is popular. He asks 10 people at the pub. 7 of them say they like it.
"70% approval," Gary announces. "Higher than any president."
Aunt Linda points out that if he asked another 10 people, he might easily get 5 yes answers. Or 9. With only 10 people, a couple of answers either way swings the result by 20 points.
This wobble from one sample to the next is called sampling error. It isn't a mistake anybody made. It's just luck in who happened to be picked. The good news: it shrinks in a completely predictable way as the sample grows.
The margin of error
Polls usually quote a margin of error, like "plus or minus 3 points". It means: our number is probably within 3 points of what we'd get if we asked absolutely everyone. ("Probably" here means 95% of the time. Lesson 9 explains exactly what that means.)
For a yes or no question where the answer is somewhere near 50/50, the margin of error depends almost entirely on the sample size:
| People asked | Margin of error (about) |
|---|---|
| 10 | plus or minus 31 points |
| 100 | plus or minus 10 points |
| 1,000 | plus or minus 3 points |
| 4,000 | plus or minus 1.5 points |
Look at the pattern. Going from 10 to 1,000 people takes you from useless to quite good. But to halve the error again, from 3 points to 1.5, you need four times as many people. The error shrinks with the square root of the sample size: four times the people gives half the error, a hundred times the people gives a tenth.
That's why polls stop at about 1,000. Asking 4,000 costs four times as much and only gains 1.5 points of accuracy. For most questions, 3 points is good enough, and the money is better spent on making sure the sample is properly random.
Margin of error for a yes or no question near 50/50
The standard error
Your lecturer will call the size of this wobble the standard error. It's the standard deviation from Lesson 3, but for the result of a sample rather than for individual values. Imagine running the same poll a thousand times with a fresh random 1,000 people each time. You'd get a thousand slightly different answers, and they'd form a bell curve (Lesson 5). The standard error is the standard deviation of that bell. The margin of error is roughly two standard errors, because 95% of a bell lies within two standard deviations of the middle.
The formula is short: standard error = the standard deviation of the data ÷ the square root of the sample size. You don't need to memorise it. You need to remember what it says: more data means less wobble, but with diminishing returns.
Size never fixes bias
Now the thought we held. All of the above assumes the sample is random. If it isn't, a bigger sample just makes you more confident about the wrong answer.
The classic example comes from the 1936 US presidential election. A popular magazine mailed out millions of ballots, mostly to people from telephone directories and car registration lists, and got about 2.4 million back. It confidently predicted that the challenger would win. Franklin D. Roosevelt won 46 of the 48 states. In 1936, people with phones and cars were, on average, much better off than everyone else, and the people who bothered to mail a ballot back weren't typical either. Both groups leaned the same way. Nearly two and a half million spoonfuls, all from the top of the pot.
Why anyone outside class cares
The same maths applies to anything measured on a sample: a drug trial, a product review score, or a trading strategy tested on a few dozen trades. A strategy that won 60% of 20 trades could easily be a coin flip that got lucky: a pure 50/50 strategy does at least that well about one time in four. Small samples produce dramatic numbers. That's why every test on this site reports how many trades it's based on.
Try it yourself
Cousin Kevin has an idea for an app. He asks 50 people chosen at random from his city, and 60% say they would pay for it. Kevin is ready to quit his job.
Using the rough rule above, what range is the true figure for the whole city probably in? And how many people would Kevin need to ask to get the margin down to about 3 points?
Show the answer
With 50 people and a result of 60%, the margin of error works out at about plus or minus 14 points. So the true figure is probably somewhere between about 46% and 74%. It could be a great idea or a coin flip. Fifty people can't tell the difference.
To get down to about 3 points, Kevin needs around 1,000 people, the same as a national poll. It doesn't matter that his city is much smaller than the country.
And one more question for Kevin: "would you pay for it?" is not the same as paying for it. That's a different kind of bias, and no sample size fixes that either.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Sample size (n) | How many people or things you measured |
| Sampling error | The natural wobble from one random sample to the next. Not a mistake, just luck |
| Sampling distribution | The bell curve you'd get if you repeated the same sample many times |
| Standard error (SE) | The standard deviation of that bell: how much a sample result typically wobbles |
| Margin of error | About two standard errors: the "plus or minus" in a poll |
| Square root law | Four times the sample gives half the error |
| Central limit theorem | The reason that bell appears: averages of random samples follow a bell curve, even when the data itself doesn't |
| Non-sampling error | Errors that a bigger sample can't fix: bias, badly worded questions, people not telling the truth |
Three things to remember
- Accuracy depends on the size of the sample, not the size of the population. A good spoonful works for any pot.
- The error shrinks with the square root of the sample size: four times the data for half the error. About 1,000 gives roughly plus or minus 3 points.
- A bigger sample never fixes a biased one. Two million wrong spoonfuls are still wrong.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.