Statistics From Zero · Lesson 21 of 22
What Is ANOVA? Comparing Three or More Averages at Once
In one sentence: ANOVA compares several group averages in one test by asking whether the differences between the groups are big compared with the ordinary variation inside them, which avoids the false alarms you'd get from running lots of t-tests.
The idea
The t-test (Lesson 14) compares two averages. But what if you have three groups, or five? You could compare every pair with its own t-test. The trouble is that every test carries its own risk of a false alarm, and the risks add up. ANOVA, short for analysis of variance, compares all the groups in a single test.
Its question is simple: are the averages further apart than you'd expect, given how much things vary anyway?
The tomato trial
Uncle Gary is convinced that his tomatoes grow best on leftover beer. Aunt Linda, who has heard this every summer for eleven years, proposes a trial. Twelve plants, three groups of four: plain water, fertiliser from the garden centre, and Gary's beer.
At the end of the summer they count the tomatoes on each plant:
| Group | Tomatoes per plant | Average |
|---|---|---|
| Plain water | 8, 10, 12, 10 | 10 |
| Shop fertiliser | 12, 14, 16, 14 | 14 |
| Gary's beer | 10, 12, 14, 12 | 12 |
Gary points out that the beer beat the water. Linda points out that the shop fertiliser beat the beer. Both are true about these twelve plants. The question is what's true about tomatoes.
Why not just run three t-tests?
Three groups make three pairs: water vs fertiliser, water vs beer, fertiliser vs beer. Each t-test at the usual 5% level has a 5% chance of a false alarm when there's no real difference. Run three, and the chance that at least one raises a false alarm grows to about 14%. With five groups there are ten pairs, and the risk reaches about 40%. This is the multiple testing problem from Lesson 8, arriving through the garden gate.
ANOVA asks one question first: is there any difference at all? Only if the answer is yes do you go looking for which.
Signal over noise: the F-ratio
ANOVA compares two kinds of variation:
- Between the groups: how far the group averages (10, 14 and 12) are from the overall average (12). This is the signal: the possible effect of what you fed the plants.
- Within the groups: how much plants that got the same treatment differ from each other (8 to 12 on plain water). This is the noise: soil, sunshine, slugs, luck.
Divide the first by the second (each scaled for the number of groups and plants) and you get the F-ratio. If the treatments did nothing, the group averages would differ only by the same luck that makes plants differ within a group, and F would be around 1. The bigger F is, the harder it is to explain the gaps as luck.
For the tomatoes, F = 6.0. With 3 groups and 12 plants, a value that high would turn up by chance only about 2% of the time: p = 0.02. Something differs.
Same averages, different verdict
Now imagine the plants within each group had varied much more, say 4, 10, 16 and 10 on plain water, with the other groups spread just as widely. The averages are still 10, 14 and 12. But the noise is now much bigger than the signal: F = 0.67, p = 0.54. No evidence of any difference.
That's the heart of ANOVA, and it echoes Lesson 3: an average without its spread is half an answer. A 4-tomato gap means a lot when plants are consistent and very little when they're all over the place.
So which group is different?
A significant ANOVA only says "at least one average differs". It doesn't say which. For that you run a follow-up test (also called a post-hoc test, Tukey's test being the common one), which compares the pairs while keeping the overall false-alarm risk at 5%.
For the tomatoes, it finds that the fertiliser beat the plain water (a gap of 4 tomatoes, p = 0.02). The beer's 2-tomato gap from either side is too small to call with four plants per group (p = 0.25 for both). So the trial shows that fertiliser helps. It doesn't show that beer helps, and it doesn't show that beer doesn't help. With four plants per group, a 2-tomato effect would be hard to catch even if it were real (Lesson 15).
Gary has announced a larger trial next summer. Linda has noted that it will require more beer.
The fine print
- Independent observations. Twelve separate plants, not one plant counted twelve times.
- Roughly bell-shaped data in each group, or enough data for the central limit theorem (Lesson 17) to help.
- Similar spread in each group. If one group varies far more than the others, use a version that allows for it (Welch's ANOVA, the cousin of Welch's t-test).
- Random assignment if you want to talk about causes. If the beer plants also happened to be in the sunniest corner, ANOVA can't tell sunshine from beer (Lesson 7).
Try it yourself
Cousin Kevin compares how many minutes three pizza places take to deliver, with five orders from each. The averages are 30, 32 and 35 minutes. ANOVA gives p = 0.30.
1. Kevin says: "So all three places are equally fast." Is that what the result shows? 2. Kevin then runs t-tests on all three pairs and reports the one with p = 0.04. What's wrong with that?
Show the answer
1. No. p = 0.30 means the gaps are the kind that ordinary variation produces quite often, so there's no evidence of a difference. That's not proof that there's no difference. With five orders each, a real 5-minute gap could easily hide in the noise. "We didn't find a difference" and "there is no difference" are different sentences.
2. That's exactly the trap ANOVA exists to avoid. After the overall test found nothing, digging through the pairs and reporting the best-looking one is multiple testing: with three tries, a p-value under 0.05 turns up by chance far more often than 5% of the time. If he wants pairwise comparisons, he should use a follow-up test that corrects for this, and expect it to agree with the ANOVA.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| One-way ANOVA | Comparing the averages of three or more groups that differ in one thing (here, what the plants got) |
| Factor, levels | The thing you change (feed) and its versions (water, fertiliser, beer) |
| Between-group variation (SSB, MSB) | How far the group averages are from the overall average: the signal |
| Within-group variation (SSW, MSW, error) | How much individuals vary inside each group: the noise |
| F-statistic (F-ratio) | Between-group variation divided by within-group variation; around 1 if nothing is going on |
| Degrees of freedom (2 and 9 here) | Groups minus 1, and observations minus groups; they set which F-table row to use |
| Post-hoc test (Tukey HSD) | A follow-up test that compares pairs without inflating the false-alarm risk |
| Family-wise error rate | The chance of at least one false alarm across a whole set of tests |
Three things to remember
- Many t-tests mean many chances of a false alarm. ANOVA asks one question first: is there any difference at all?
- F is signal over noise: gaps between groups compared with variation within them.
- A significant ANOVA says something differs, not what. A non-significant one says "no evidence", not "no difference".
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.