Statistics From Zero · Lesson 12 of 22
Linear Regression: Drawing the Line (and Knowing Where to Stop)
In one sentence: Regression draws the straight line that best fits your data and tells you how much one thing tends to change when another does, but it only knows about the range of data you gave it.
The idea
In Lesson 7 you met correlation: a single number saying how tightly two things move together. Regression goes one step further. It draws the actual line through the data, so you can say how much one thing changes when the other does, and make a prediction.
The line it draws is the one that keeps the points as close to it as possible overall. More precisely, it makes the squared vertical distances from the points to the line as small as possible, which is why it's called least squares. You never need to draw it by hand; every spreadsheet does it. What you need is to know what the line can and can't tell you.
Cousin Kevin's lemonade stand
Cousin Kevin runs a lemonade stand at the summer market. He wrote down the temperature and the cups he sold on five Saturdays:
| Temperature | 60°F | 65°F | 70°F | 75°F | 80°F |
|---|---|---|---|---|---|
| Cups sold | 20 | 26 | 29 | 35 | 40 |
The best-fitting straight line through these points is:
cups = 0.98 × temperature - 38.6
Two numbers, two meanings:
- The slope, 0.98: each extra degree goes with about one extra cup sold. This is the useful number. It's the "how much".
- The intercept, -38.6: the line's value at 0°F. Here it means nothing at all (Kevin can't sell minus 38 cups). It's just where the line has to start so that it passes through the right place.
For a forecast of 85°F, the line predicts 0.98 × 85 - 38.6 = about 45 cups. Kevin bought lemons for 45. Reasonable.
How good is the line?
The points don't sit exactly on the line. The gaps are called residuals: at 65°F Kevin sold 26 cups and the line said 25.1, so the residual is +0.9. Small residuals mean a good fit.
The usual summary is R squared: the share of the ups and downs in sales that the line explains. Here it's 0.99, meaning temperature explains 99% of the variation in Kevin's sales. That's suspiciously tidy, because these are made-up numbers. Real data is messier, and an R squared of 0.3 can still be useful, while 0.99 in real life usually means you've made a mistake or you're measuring the same thing twice.
Where to stop
Uncle Gary borrowed Kevin's line to plan a winter market stall. At 32°F, freezing point, the line predicts 0.98 × 32 - 38.6 = minus 7 cups.
Gary took this to mean that customers would bring him lemonade. He set up two empty buckets. Nobody came, in either direction.
This is extrapolation: using the line outside the range of the data that built it. Kevin's data went from 60 to 80°F. Between those, the line is a good guide. Outside them, it has no idea. Maybe sales stop falling below 50°F because the regulars always come. Maybe at 100°F everyone stays home. The line can't know, because it never saw those days.
Markets fall into this trap constantly. A straight trend line drawn through the last three good years tells you nothing about the fourth, and a strategy fitted to a calm period says nothing about a crash. That's why our tests check results on data the rules were never fitted to.
Two more warnings
- A line is not a cause. Everything from Lesson 7 still applies. Kevin's sales and sunburn cases both rise with temperature; a regression of one on the other would fit well and mean nothing.
- Regression to the mean is a different thing with a confusingly similar name. It's the tendency for extreme results to be followed by more ordinary ones, like a brilliant employee of the month having a normal month next. The word "regression" in both comes from the same 19th-century study of heights, which is the only reason they share it.
Try it yourself
A study of Kevin's market finds: daily visitors = 12 × number of stalls + 150, based on days with 10 to 40 stalls.
1. What does the 12 mean? 2. How many visitors does the line predict with 25 stalls? 3. The organiser wants to know how many visitors 200 stalls would bring. What do you tell them?
Show the answer
1. Each extra stall goes with about 12 extra visitors a day (on average, within the data).
2. 12 × 25 + 150 = 450 visitors.
3. The line says 2,550, but 200 stalls is far outside the 10 to 40 the data covered. The car park might fill up, the town might not have that many shoppers, or visitors might spread out across more stalls instead of increasing. It's extrapolation; the honest answer is "we don't know, and the line can't tell us". And even inside the range, more stalls might follow more visitors rather than cause them.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Linear regression | Fitting the best straight line through the data |
| Dependent variable (y) | What you're trying to predict (cups sold) |
| Independent variable (x), predictor | What you predict it from (temperature) |
| Slope (coefficient, b) | How much y changes, on average, when x goes up by one |
| Intercept (a) | The line's value when x is zero; often meaningless on its own |
| Least squares | The method: choose the line that makes the squared gaps as small as possible |
| Residual | The gap between a real point and the line's prediction |
| R squared | The share of the variation in y that the line explains, from 0 to 1 |
| Extrapolation | Using the line outside the range of the data. Dangerous |
Three things to remember
- The slope is the useful number: how much one thing changes when the other goes up by one.
- A line only knows the range of data it was built on. Beyond it, it's guessing with confidence.
- A good fit is not a cause, and a perfect fit in real data is a reason for suspicion.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.