Statistics From Zero · Lesson 25 of 25
Observational Study vs Experiment: How to Know What Causes What
In one sentence: In an observational study you only watch what people already do, so hidden differences can explain the result; in an experiment you assign the treatment at random, which is the cleanest way to show cause and effect.
The idea
Lesson 7 warned that correlation isn't causation. This lesson is about what you do instead. There are two basic ways to collect data about a cause:
- Observational study: you watch and record what people, plants or companies already do, and compare the groups. You don't change anything.
- Experiment: you decide who gets the treatment, ideally at random, and compare the results.
The difference sounds small. It's the difference between "these things go together" and "this causes that".
The gnome theory
Uncle Gary has walked up and down the street and made a discovery: the houses with garden gnomes have noticeably better lawns. His conclusion is that gnomes improve lawns, possibly through morale. He has ordered six.
Aunt Linda points out that most of the gnome owners are retired and spend all day on their lawns.
Gary ran an observational study. The gnome houses and the no-gnome houses differ in many ways besides gnomes: free time, age, money, interest in gardening. Any of those could explain the lawns. These hidden differences are confounding variables (Lesson 7), and in an observational study you can never be sure you've thought of them all.
What random assignment does
Now imagine Linda's version. She takes 40 volunteer households, flips a coin for each, and gives gnomes to the 20 that get heads. A summer later she compares the lawns.
The coin doesn't care who is retired, who is rich or who loves gardening. On average, those things end up spread evenly between the two groups. So if the gnome group's lawns end up better, the gnomes (or something that comes with receiving a gnome) are the most plausible reason. That's the power of random assignment: it balances the confounders you know about and the ones you don't.
The no-gnome households are the control group: they show what would have happened anyway, including weather, season and regression to the mean (Lesson 23).
The other ingredients of a good experiment
- A fair comparison: both groups treated the same in every way except the thing being tested. If the gnome households also get a visit from an enthusiastic Gary every week, you're testing Gary, not gnomes.
- Blinding where possible: the people judging the lawns shouldn't know which houses have gnomes, so their expectations can't colour the scores. If the households themselves don't know whether they're in the "real" group, the test is double-blind.
- Enough cases: twenty lawns a side can only catch a fairly big effect (Lessons 15 and 22).
- Decide the measure in advance: "lawn quality on a 1 to 10 scale in September", not whatever looks best afterwards (Lesson 8's multiple testing).
When you can't experiment
Many questions can't be answered with a coin flip: you can't randomly assign people to smoke, to be born in a certain year, or to go through a recession. Then observational studies are all you have, and careful ones are still very valuable. They can compare like with like (Lesson 18), adjust for known confounders, follow the same people over time, and look for natural experiments, situations where chance did the assigning for you, such as a lottery deciding school places. The honest wording for an observational result is "is associated with", not "causes".
Why this matters for trading results
A backtest is an observational study of the past: we can't rerun history with the signal randomly switched on and off. That's why our tests always compare a strategy with a control group of random stock picks over the same period, and why we're careful with what a result can show. A strategy that looked good in the past is associated with good results then; that's a much weaker statement than "it causes profits".
Try it yourself
(A made-up example.) A survey of a school finds that students who choose to sit in the front row average 72 in exams and students at the back average 64. Gary concludes that sitting at the front adds 8 points. The next year, a teacher assigns seats by drawing names from a hat. Front-row students average 69 and back-row students 67.
1. Which study is observational and which is an experiment? 2. Why did the gap shrink? 3. What's the best estimate of the effect of sitting at the front?
Show the answer
1. The survey is observational: students chose their own seats. Drawing names from a hat is an experiment with random assignment.
2. Students who choose the front row are probably more motivated to begin with. In the survey, the 8-point gap mixes the seat effect with motivation. The hat spread motivated students evenly across the room, so most of the gap disappeared.
3. The experiment's 2 points, with a caveat: one class in one year is a small sample, so it needs a range (Lesson 9) before anyone rearranges the furniture. Gary has moved to the front row of the family dinner table, without any measurable effect.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Observational study | Recording what happens without assigning any treatment |
| Experiment (randomised controlled trial, RCT) | The researcher assigns the treatment, ideally at random, and compares with a control group |
| Treatment group, control group | Those who get the thing being tested, and those who don't |
| Random assignment (randomisation) | Using chance to decide who gets the treatment, so confounders balance out on average |
| Confounding variable | A hidden difference between groups that affects the outcome |
| Blinding, double-blind | Keeping judges (and participants) unaware of who got the treatment |
| Natural experiment | A real-world situation where chance assigned the "treatment" |
| Random sampling vs random assignment | Sampling decides who is studied (Lesson 1); assignment decides who gets the treatment |
Three things to remember
- Watching shows what goes together. Assigning at random shows what causes what.
- Random assignment balances the confounders you know about and the ones you haven't thought of.
- When you can only observe, say "is associated with", compare like with like, and look for a control group.
Part 3 so far
Lessons 20 to 25 add tools you'll meet in almost every course and every research report: the binomial and Poisson distributions for counting, ANOVA for comparing more than two groups, effect size for deciding whether a significant result matters, regression to the mean for spotting changes that need no cause, and the difference between watching and experimenting for deciding what causes what.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.