Statistics From Zero · Lesson 9 of 22
Between Two and Six Pints: Confidence Intervals
In one sentence: A single number from a sample is almost certainly a bit wrong, so a good estimate comes with a range, and the width of that range tells you how much the number is worth.
The idea
When you estimate something from a sample, you get one number: "the average student drinks 4 pints on a night out". That number is almost certainly a little off, because of the sampling wobble from Lesson 6. Ask a different group and you'd get 3.6, or 4.5.
So instead of pretending the number is exact, statisticians give a range: "somewhere between 2 and 6 pints". That range is a confidence interval. It's the honest version of an estimate. The number in the middle is your best guess; the width of the range tells you how seriously to take it.
Cousin Kevin's pint survey
Kevin is writing a university essay about student life. He asks 9 of his friends how many pints they drink on a typical night out. The average answer is 4, with a standard deviation of 3. (Kevin's friends vary. Some are on the rugby team. One is called Martin and drinks lemonade.)
Kevin writes: "Students drink 4 pints on a night out."
His tutor writes in the margin: "Between about 2 and 6, from your data. Also, these are your friends."
The tutor is right on both counts. The second one is bias (Lesson 1), and no interval fixes that. The first one is the confidence interval, and here's where it came from.
How the range is built
You already have every piece from earlier lessons:
- The estimate: the sample average, 4 pints.
- The standard error (Lesson 6): standard deviation ÷ square root of the sample size. Here that's 3 ÷ 3 = 1 pint.
- The bell curve rule (Lesson 5): about 95% of a bell lies within 2 standard deviations of the middle. (The precise number is 1.96, which everyone rounds to 2.)
Put together: estimate plus or minus about 2 standard errors. 4 plus or minus 2 gives roughly 2 to 6 pints. That's a 95% confidence interval.
Want to be more sure? A 99% interval uses about 2.6 standard errors instead of 2, so it's wider: roughly 1.4 to 6.6. More confidence always costs you a wider range. The only way to get a range that's both narrow and trustworthy is more data.
If Kevin asked 36 people instead of 9 (four times as many), the standard error would halve to 0.5, and the 95% interval would shrink to roughly 3 to 5 pints. The square root law from Lesson 6, in action.
(For very small samples like Kevin's 9, the exact calculation uses a slightly wider multiplier than 2. That's what the "t-distribution" in your textbook is for. The idea is exactly the same.)
What "95% confident" actually means
This is where most people, and plenty of textbooks, get slightly muddled. The true average for all students is a fixed number. It's either inside Kevin's interval or it isn't. So "95%" isn't really about this one interval.
It's about the method. If you repeated the survey many times, each time with a fresh random sample, and built an interval the same way each time, about 95% of those intervals would contain the true value. About 1 in 20 would miss it completely, through no fault of anyone.
Think of it as a fishing net. A 95% net catches the fish 95% of the time. Once you've thrown it, the fish is either in this particular net or it isn't, but you used a net that rarely misses.
Twenty repeated surveys, twenty 95% intervals
Intervals and significance are the same idea
Confidence intervals and the hypothesis tests from Lesson 8 are two views of the same maths. If a 95% interval for a difference doesn't include zero, the difference is statistically significant at 5%. If it does include zero, it isn't. The interval just shows you more: not only whether the effect might be zero, but how big it plausibly is.
Try it yourself
Two trading strategies were each tested on historical data. Results are the average profit per trade, with a 95% confidence interval. (These are made-up examples for the exercise.)
Strategy A: average +0.4% per trade, 95% interval from minus 0.2% to +1.0%.
Strategy B: average +0.10% per trade, 95% interval from +0.06% to +0.14%.
Trading costs are about 0.05% per trade (buying and selling together). Which strategy, if either, has shown it makes money?
Show the answer
Strategy A has the exciting average, but its interval includes zero, and some negative numbers too. The data is consistent with a strategy that loses money. Not significant. It needs far more trades before anyone can say anything.
Strategy B is reliably above zero: the whole interval is positive, so it's statistically significant. But now subtract the costs. 0.10% minus 0.05% leaves 0.05% per trade, and the low end of the interval, 0.06%, leaves just 0.01%. Real, but tiny, and one slightly worse assumption about costs wipes it out.
So neither has convincingly shown it makes money after costs. A is a big maybe. B is a small yes that becomes a "probably not worth it" after costs. That's very typical of what we find when we test trading ideas, and exactly why we always show the range, the number of trades and the costs, not just the average.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Point estimate | The single best guess (the 4 pints) |
| Interval estimate | A range instead of a single number |
| Confidence interval (CI) | The range built as estimate plus or minus a few standard errors |
| Confidence level | How often the method catches the true value: usually 95%, sometimes 90% or 99% |
| Margin of error | Half the width of the interval: the "plus or minus" part |
| Critical value | The multiplier: about 1.96 for 95%, about 2.58 for 99% |
| t-distribution | A slightly wider version of the bell curve, used for small samples |
| Coverage | The share of intervals that would contain the true value if you repeated everything |
Three things to remember
- A good estimate always comes with a range. A single number with no range is hiding how unsure it is.
- A 95% interval is roughly the estimate plus or minus 2 standard errors. More data makes it narrower; more confidence makes it wider.
- If the interval includes zero, the effect might not exist. If it's entirely above zero but tiny, ask whether it survives costs.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.