Statistics From Zero · Lesson 28 of 31
The Bootstrap: A Confidence Interval From Your Own Data
In one sentence: The bootstrap estimates how much a result would vary from sample to sample by resampling your own data with replacement thousands of times, which gives a confidence interval even when no formula exists.
The idea
A confidence interval (Lesson 9) needs to know how much your result would wobble if you took a different sample. For an average there's a formula for that wobble, the standard error. For many other things (a median, a percentile, a ratio, anything odd) there's no simple formula.
The bootstrap gets around it with a trick: treat your sample as a stand-in for the whole population, draw lots of new samples from your own data, and see how much the result varies between them. That variation is your estimate of the wobble.
Cousin Kevin's commute
Cousin Kevin has a new job and has timed his first 15 commutes, in minutes: 22, 25, 26, 27, 28, 29, 30, 31, 33, 35, 38, 41, 47, 58, 75. The 75 was a day with a broken-down train.
He wants to tell his boss how long the commute "usually" takes, with a range. Uncle Gary says he should just quote the worst day "to be safe". Aunt Linda says the median is 31 minutes, but how sure can he be after only 15 days?
The data are right-skewed (Lesson 26), so the median, 31, describes a typical day better than the mean, 36.3. But Lesson 9's formula is for means. How do you put a range around a median?
How the bootstrap works
- Step 1. Resample. Draw 15 values from Kevin's 15, with replacement: after picking a value, put it back, so it can be picked again. One such resample might be 25, 26, 26, 27, 28, 28, 29, 30, 30, 33, 35, 38, 41, 47, 75. Some days appear twice, some not at all.
- Step 2. Calculate. Work out the median of the resample. Here it's 30.
- Step 3. Repeat. Do it thousands of times. A computer does 100,000 in a second.
- Step 4. Read off the range. Sort all the resampled medians. The middle 95% of them, from the 2.5th to the 97.5th percentile, is a 95% bootstrap confidence interval.
Why with replacement? Without it, every resample would be Kevin's 15 days in a different order, with exactly the same median. Putting values back mimics drawing a fresh sample from a population that looks like his data.
Kevin's answer
We ran 200,000 resamples. The medians bunched around 30 and 31 and spread out from there:
| Median of the resample | Share of resamples |
|---|---|
| 27 or less | 2.5% |
| 28 to 38 | 95% |
| 41 or more | 2.5% |
So a 95% interval for Kevin's typical commute is roughly 27 to 41 minutes, centred on 31. With 15 days, that's as precise as he can honestly be. He can tell his boss "usually about half an hour, somewhere between 27 and 41 on a typical day", which is more useful than Gary's 75.
Medians of 200,000 bootstrap resamples of 15 commutes
Does it agree with the formula?
For the mean, where a formula exists, we can check. The bootstrap gives a 95% interval of 30.1 to 43.9 minutes. The usual formula, the mean plus or minus about two standard errors using the t-distribution (Lesson 14), gives 28.5 to 44.2. Close, and the bootstrap's is slightly lopsided, following the skew of the data. The bootstrap's estimate of the standard error (3.56) is also close to the formula's (3.67). When both methods are available, they usually agree; the bootstrap's value is that it keeps working where the formula stops.
Where it fails
- Very small samples. With five values there's too little data to stand in for the population, and the interval will be too narrow or oddly shaped.
- Extremes. A resample can never contain a value larger than the largest in your data. 64% of Kevin's resamples had the same maximum, 75 minutes, so the bootstrap can't say anything sensible about his worst possible commute.
- Dependent data. Resampling single days breaks any pattern over time, such as one bad week of roadworks. Data that come in runs (including market prices) need special versions that resample blocks.
- A biased sample stays biased. If Kevin only timed the days he left early, resampling them a million times won't fix it (Lesson 1).
Try it yourself
A café owner records the number of customers on 30 days and wants a 95% interval for the 90th percentile (a busy day). She bootstraps 10,000 resamples and finds that the 90th percentile of the resamples ranges, in its middle 95%, from 61 to 78 customers.
1. What should she conclude? 2. Why would the bootstrap be a poor tool for the busiest day she will ever have?
Show the answer
1. A typical busy day (the 90th percentile) is probably somewhere between about 61 and 78 customers; that's the range to staff for. The interval is wide because 30 days contain only three days above the 90th percentile.
2. The bootstrap can't produce a day busier than the busiest in her 30 days, so it says nothing about rare, extreme days. For those she needs a longer record, or a model of the tail.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Bootstrap | Estimating the variation of a result by resampling the data many times |
| Resampling with replacement | Drawing values and putting each back, so it can be drawn again |
| Bootstrap sample (resample) | One new sample of the same size drawn from your data |
| Bootstrap distribution | The spread of the result across all the resamples |
| Percentile interval | The middle 95% of the bootstrap distribution, used as a 95% confidence interval |
| Bootstrap standard error | The standard deviation of the bootstrap distribution |
| Block bootstrap | A version that resamples runs of consecutive values, for data over time |
| Nonparametric | Not assuming a particular shape, such as the bell curve, for the data |
Three things to remember
- Resample your own data with replacement, recompute the result each time, and take the middle 95%.
- It works for medians, percentiles and other results with no simple formula, and agrees with the formula when there is one.
- It can't invent what isn't in the sample: no values beyond your extremes, no fix for bias, and care is needed with tiny or time-ordered data.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.