Statistics From Zero · Lesson 16 of 22
The Chi-Square Test: Is the Die Loaded?
In one sentence: The chi-square test compares the counts you got with the counts you'd expect if nothing interesting were going on, and tells you whether the gap is bigger than chance usually produces.
The idea
The t-test (Lesson 14) compares averages. But a lot of data isn't averages at all, it's counts: how many sixes, how many people chose each option, how many rainy days after a sore knee. For counts, the standard tool is the chi-square test (chi is the Greek letter χ, pronounced "kai").
The idea is simple. Work out what the counts should look like if nothing interesting is going on. Compare them with what you actually got. Add up the gaps in a clever way. If the total is bigger than chance usually produces, something interesting probably is going on.
The suspicious die
At the summer fair, the stall with the dice game (Lesson 13) has been paying out a lot of sixes, at least to other people. Uncle Gary is convinced the stall's die is loaded. He borrowed it, rolled it 60 times and counted:
| Face | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Rolled (observed) | 8 | 9 | 7 | 10 | 9 | 17 |
| Fair die (expected) | 10 | 10 | 10 | 10 | 10 | 10 |
Seventeen sixes instead of ten. Gary is ready to call the police. Let's do the test first.
Computing it by hand
For each face, take (observed - expected), square it, and divide by the expected count. Then add them up:
- Face 1: (8 - 10)² ÷ 10 = 0.4
- Face 2: (9 - 10)² ÷ 10 = 0.1
- Face 3: (7 - 10)² ÷ 10 = 0.9
- Face 4: (10 - 10)² ÷ 10 = 0
- Face 5: (9 - 10)² ÷ 10 = 0.1
- Face 6: (17 - 10)² ÷ 10 = 4.9
- Total: χ² = 6.4
Squaring makes every gap positive and punishes big gaps more than small ones; dividing by the expected count puts gaps on a fair scale.
Now, is 6.4 a lot? It depends on how many categories could vary freely, the degrees of freedom: here 6 faces minus 1 = 5 (once you know five counts and the total, the sixth is fixed). With 5 degrees of freedom, a fair die produces a χ² above 11.07 only 5% of the time. Gary's 6.4 is well below that. The p-value is about 0.27: a fair die would give a result at least this lopsided about 27 times in 100.
Verdict: no real evidence that the die is loaded. Seventeen sixes in 60 rolls looks dramatic, but fair dice do that kind of thing surprisingly often.
When the same pattern becomes proof
Now suppose Gary had the patience to roll it 600 times and got exactly the same proportions: 80, 90, 70, 100, 90 and 170 sixes. The expected count per face is 100, and χ² comes out at 64, with a p-value below one in a million. Same pattern, ten times the data: now it's overwhelming. Exactly the lesson of power from Lesson 15. With 60 rolls, the test could barely have caught a loaded die; with 600 it can't miss one.
The other chi-square test: are two things related?
The same idea works for tables. In our Grandpa's knee story, 100 example days were sorted by two things: did his knee ache, and did it rain the next day?
| Rain next day | Dry | |
|---|---|---|
| Knee aches (40 days) | 12 | 28 |
| Knee quiet (60 days) | 18 | 42 |
If the knee and the rain are unrelated, you'd expect the 30% rain rate to apply to both rows: 12 of 40 and 18 of 60. That's exactly what happened, so χ² = 0: no relationship at all. This is the chi-square test of independence, and it's the standard way to check whether two categories go together.
Try it yourself
Aunt Linda asked 90 family members which of three dates they preferred for Christmas lunch. Results: 24th: 40, 25th: 30, 26th: 20. If the family had no preference, you'd expect 30 each.
1. Compute χ². 2. With 2 degrees of freedom, the 5% bar is 5.99. Is there a real preference?
Show the answer
1. (40 - 30)² ÷ 30 + (30 - 30)² ÷ 30 + (20 - 30)² ÷ 30 = 3.33 + 0 + 3.33 = 6.67.
2. 6.67 is above 5.99, so yes: the preference for the 24th is bigger than chance usually produces (the p-value is about 0.036). Gary, who voted for the 26th, has requested a recount. And remember the group chat lesson: the test assumes these 90 answers represent the family. If only the eager relatives answered, the chi-square is precise about the wrong people.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Chi-square (χ²) test | A test that compares observed counts with expected counts |
| Goodness of fit | The version that checks whether counts match a specific expectation (a fair die) |
| Test of independence | The version that checks whether two categories are related (knee and rain) |
| Observed (O) and expected (E) frequencies | The counts you got, and the counts you'd get if nothing interesting were happening |
| Degrees of freedom | How many counts can vary freely; for one list, the number of categories minus one |
| Contingency table | A table of counts sorted by two categories at once |
| Expected count of at least 5 | The usual rule for when the test is reliable; with tiny counts it isn't |
Three things to remember
- Chi-square compares the counts you got with the counts you'd expect from chance.
- A lopsided-looking table can easily be chance in a small sample; the same pattern in a large sample can be overwhelming.
- The test is only as good as the counting: who was counted matters as much as the arithmetic.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.