Statistics From Zero · Lesson 27 of 31
Multiple Regression: What "Controlling For" Really Means
In one sentence: Multiple regression estimates the link between one factor and an outcome while holding other measured factors fixed, which can make a misleading link disappear, but it can't fix factors nobody measured.
The idea
In Lesson 12 we drew a line through a cloud of points: one cause, one outcome. Real life usually has several things going on at once. Multiple regression fits one equation with several explanatory factors together, and gives each one its own number: how much the outcome changes when that factor goes up by one, with the other factors in the model held fixed.
That last phrase is what researchers mean by "controlling for" something. It's the main tool for dealing with confounding variables (Lesson 7) when you can't run an experiment (Lesson 25).
Uncle Gary's coffee theory
Uncle Gary has read a fitness survey. "People who drink the most coffee have the worst fitness scores," he announces, pouring his fourth cup down the sink with great ceremony. "Coffee ruins your lungs."
Aunt Linda points out that half of the heavy coffee drinkers she knows take their coffee outside with a cigarette.
To see what's going on, we built a made-up survey of 1,000 people where we know the truth: 30% smoke, smokers drink about two cups of coffee a day more than non-smokers, smoking lowers the fitness score (out of 100) by 15 points, and coffee has no effect at all. Then we analysed it two ways, and repeated the whole thing 2,000 times to get stable numbers.
| Model | Effect of one extra cup of coffee | Effect of smoking |
|---|---|---|
| Simple regression: fitness on coffee | 3.5 points lower | not in the model |
| Multiple regression: fitness on coffee and smoking | 0.0 points | 15 points lower |
With coffee alone, the line slopes down, and it's not a fluke: heavy coffee drinkers really do score lower. But that's because heavy coffee drinkers are mostly smokers. Add smoking to the model and the coffee number drops to zero, because now it's answering a different question: among people with the same smoking status, does more coffee go with lower fitness? It doesn't.
What "holding fixed" looks like
You can see the same thing without any equations by splitting the people into groups:
| Average fitness score | Under 2 cups a day | 4 cups or more |
|---|---|---|
| Everyone | 79.7 | 66.4 |
| Non-smokers only | 80.0 | 79.9 |
| Smokers only | 65.1 | 65.0 |
The 13-point gap in the first row comes entirely from who is in each column: only 2% of the light coffee drinkers smoke, against 91% of the heavy ones. Inside each smoking group, coffee makes no difference. Multiple regression does this splitting for you, all at once, for many factors, including ones like age or income that can't be split into two neat groups. It's a close relative of Simpson's paradox (Lesson 18).
The model with both factors also explains more: R squared (Lesson 12) rose from about 0.23 to about 0.46, because smoking is the factor that actually drives the scores.
Average fitness score by coffee drinking, all and by smoking
How to read the output
- Each coefficient is the expected change in the outcome for one more unit of that factor, with the other factors in the model unchanged.
- Each comes with a standard error and a p-value (Lessons 8 and 14): is it distinguishable from zero?
- The sign can flip when you add a factor. That's not a bug; it's the point.
Three ways "controlling for" goes wrong
- You can only control for what you measured. If the survey hadn't asked about smoking, the coffee effect would stay at minus 3.5 and look convincing. Regression can't see a confounder that isn't in the data. That's why experiments still win.
- Controlling for the wrong thing. If coffee affected fitness through sleep, adding sleep to the model would hide a real effect. Don't control for steps along the path you're trying to measure.
- Factors that move together. If two factors are almost the same thing (height in centimetres and in inches), the model can't tell them apart, and their coefficients become wild and unreliable.
Try it yourself
A study finds that people who own more books earn more. After controlling for parents' income and years of education, the link becomes much smaller but doesn't vanish.
1. Why did the link shrink? 2. Does what's left prove that buying books raises your salary?
Show the answer
1. Part of the original link came from confounders: richer parents and more education go with both more books and higher pay. Holding those fixed removes their share.
2. No. Other unmeasured factors (curiosity, health, where people live) could still explain the remaining link, and the cause could even run the other way: higher earners can afford more books. Controlling makes a causal story more plausible; it doesn't prove it.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Multiple regression | A regression with two or more explanatory factors at once |
| Dependent variable (outcome) | The thing you're trying to explain, such as the fitness score |
| Independent variables (predictors, covariates) | The factors in the model, such as coffee and smoking |
| Partial regression coefficient | The effect of one factor with the others held fixed |
| Controlling for, adjusting for | Including a factor in the model so its influence is held fixed |
| Omitted variable bias | A wrong coefficient caused by leaving out a confounder |
| Mediator | A step on the path from cause to outcome; controlling for it hides the effect |
| Multicollinearity | Factors so closely related that the model can't separate their effects |
Three things to remember
- Each coefficient answers: what changes with this factor when the others in the model stay the same?
- Adding a confounder can shrink, erase or flip a link. That's the method working.
- Regression can only hold fixed what you measured. An unmeasured confounder still fools it.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.