Statistics From Zero · Lesson 14 of 22
The t-Test: Is the Difference Real or Just Noise?
In one sentence: A t-test compares the gap between two averages with how much the data wobbles and how much data there is, and asks whether the gap stands out from the noise.
The idea
Two groups, two averages, and a difference between them. Is the difference real, or would you get one that size by luck? That's the most common question in statistics, and the t-test is the most common way to answer it.
It weighs three things at once: how big the gap is, how much the numbers wobble within each group, and how many numbers you have. A big gap, little wobble and lots of data make a convincing difference. A small gap, lots of wobble and a handful of data make a coin toss.
Uncle Gary's shortcut
Uncle Gary has discovered a shortcut to Grandpa Frank's house. It involves a farm track, a closed petrol station and a turn he refuses to describe. He says it saves "loads of time".
Aunt Linda, who drives the normal way, suggested a test. They each timed six trips, in minutes:
| Trip times (minutes) | Average | |
|---|---|---|
| Linda's normal route | 32, 35, 30, 38, 33, 36 | 34 |
| Gary's shortcut | 30, 34, 28, 33, 31, 30 | 31 |
Gary's shortcut is 3 minutes faster on average. Gary has already told Grandpa. But look at the times: Linda's route ranged from 30 to 38 minutes, Gary's from 28 to 34. The routes overlap a lot. Is 3 minutes a real difference, or just a lucky week of traffic?
What the t-test does
The t-test turns the question into one number, the t-statistic:
t = the difference between the averages ÷ the standard error of that difference
The standard error (Lesson 6) measures how much the difference would bounce around if you repeated the whole experiment. It's bigger when the times wobble a lot and smaller when you have more trips. Here:
- The difference: 34 - 31 = 3 minutes.
- The standard error of that difference: about 1.5 minutes (from how much each route's times wobble, with six trips each).
- t = 3 ÷ 1.5 = about 2.0.
So the gap is about twice its typical bounce. The rough rule from Lesson 9: a t around 2 or more is where "probably not luck" starts. Here we're right on the edge. The exact p-value is 0.07: if the routes were really identical, a gap this big would turn up about 7 times in 100. That's suggestive, but it doesn't clear the usual 5% bar.
So who wins the argument?
Neither, yet. Gary can't say his shortcut is proven faster. Linda can't say it isn't faster: "not significant" means "not proven", not "no difference" (Lesson 8). The honest verdict is: six trips each can't tell.
The fix is more data. If they'd done twelve trips each, with the same averages and the same wobble, the standard error would shrink and t would rise to about 3.0, with a p-value around 0.007. The same 3 minutes would then be convincing. Nothing about the shortcut would have changed; only the amount of evidence.
That's the most useful thing to take from the t-test: whether a difference is "real" depends not only on how big it is but on how much noise surrounds it and how much data you have.
Which t-test?
- Two separate groups (different trips on each route): the independent-samples t-test. The version that doesn't assume both groups wobble equally is called Welch's t-test, and it's a safe default.
- The same things measured twice (the same ten drivers on both routes): the paired t-test. It compares each driver with themselves, which removes a lot of noise and usually needs less data.
- One group against a fixed number (is the average trip really under 35 minutes?): the one-sample t-test.
All three do the same thing underneath: a difference divided by its standard error.
Try it yourself
Cousin Kevin claims his new coffee machine makes the office happier. Satisfaction scores (out of 10) from 50 colleagues: old machine average 6.1, new machine average 6.4. The standard error of the difference is 0.3.
1. What's the t-statistic? 2. Is the difference convincing? 3. What would make it more convincing?
Show the answer
1. t = (6.4 - 6.1) ÷ 0.3 = 1.0.
2. No. The gap is the same size as its typical bounce, which is very easy to get by luck. Not proven, which isn't the same as "no effect".
3. More colleagues, or a paired design: ask the same people before and after, so each person's general mood cancels out. And a little suspicion: if Kevin chose the day and asked the questions himself, the bias problems from Lesson 6 matter more than the t-test.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| t-test | A test of whether a difference between averages is bigger than noise would produce |
| t-statistic | The difference divided by its standard error: how many "typical bounces" big the gap is |
| Standard error of the difference | How much the difference would vary if you repeated the experiment |
| Independent samples / Welch's t-test | Two separate groups; Welch's version doesn't assume equal spread |
| Paired t-test | The same subjects measured twice; compares each with itself |
| Degrees of freedom | A count based on the sample size that sets how strict the t bar is; with small samples, the bar is a bit higher than 2 |
| Two-sided test | Checking for a difference in either direction, not only the one you hoped for |
Three things to remember
- A t-test asks whether a difference is big compared with its own noise, not whether it's big.
- t around 2 is where "probably not luck" starts; small samples need a bit more.
- Not significant means not proven. More data, or a paired design, can change the verdict without changing the difference.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.