Statistics From Zero · Lesson 30 of 31
Rank Tests: When Your Data Won't Behave (Mann-Whitney)
In one sentence: Rank tests compare groups using the order of the values instead of the values themselves, so a single extreme value can't swamp them, which makes them the safer choice for small, skewed samples.
The idea
The t-test (Lesson 14) compares averages, and averages are sensitive: one extreme value can move them a lot (Lessons 2 and 26). A rank test throws away the exact values and keeps only their order: smallest, second smallest and so on. A value of 95 or 9,500 gets the same rank if it's the biggest. That makes rank tests robust to outliers and skewed data, at the cost of a little power when the data are well behaved.
The most common one for two groups is the Mann-Whitney U test (also called the Wilcoxon rank-sum test).
The pizza race
Uncle Gary's stopwatch is back. This time he timed seven deliveries from each of the two pizza places in town. Roma: 21, 22, 24, 25, 26, 27 and 95 minutes (the 95 involved a wrong address and a dog). Napoli: 29, 30, 31, 32, 34, 35 and 36 minutes.
"Roma averages 34 minutes, Napoli 32," says Gary. "Napoli is faster." Aunt Linda says Roma was faster on six of the seven nights.
Cousin Kevin runs a t-test on the two groups: t = 0.18, p = 0.86. No difference at all, apparently. But look at the numbers again: every Roma delivery except one beat every Napoli delivery. The t-test is fooled by the single 95, which inflates both Roma's average and its spread.
Delivery times in minutes, seven orders from each place
How the rank test sees it
- Step 1. Pool and rank. Put all 14 times in order, ignoring which place they came from. Roma's 21 gets rank 1, 22 rank 2, and so on; Roma's six quick deliveries take ranks 1 to 6, Napoli's take 7 to 13, and the 95 takes rank 14.
- Step 2. Count the wins. Compare every Roma delivery with every Napoli delivery: 7 × 7 = 49 pairs. Roma was quicker in 42 of them and slower in 7 (the 95 lost all seven of its pairs). The smaller count, 7, is the test statistic U.
- Step 3. Ask how surprising that is. If the two places were really the same, the 14 ranks would be shuffled at random between them. There are 3,432 ways to split 14 ranks into two groups of 7, and only a small fraction give a split as lopsided as this one. The exact p-value is 0.026.
So the rank test says, with reasonable confidence, that Roma tends to be quicker, while the t-test said nothing. Neither is "wrong"; they ask different questions. The t-test asks about averages, and Roma's average really is dragged up by one disaster. The rank test asks whether a random Roma delivery tends to beat a random Napoli one, which is closer to what Gary actually wants to know.
To see how much the one value mattered: change the 95 to 28 and the t-test also becomes clearly significant (t = -5.55, p below 0.001).
What the Mann-Whitney test actually tests
It's often described as a "test of medians", but strictly it tests whether values from one group tend to be larger than values from the other: the chance that a random pick from group A beats a random pick from group B. When the two groups have similar shapes, that's the same as comparing medians. When their shapes differ a lot, report the "chance of beating" instead.
When to use a rank test
- Small samples with skew or outliers, where you can't count on the central limit theorem (Lesson 17) to rescue the t-test.
- Ordered categories, such as ratings from 1 to 5 stars, where the distance between 4 and 5 isn't really the same as between 1 and 2.
- Not when you need the average itself, such as total costs or total profits. If one disaster really costs money, the average is the right thing to compare, outlier and all.
There are rank versions of other tests too: the Wilcoxon signed-rank test for paired data (the rank cousin of the paired t-test) and the Kruskal-Wallis test for three or more groups (the rank cousin of ANOVA, Lesson 21).
Try it yourself
Two groups of five customers rate a new menu from 1 (awful) to 5 (great). Group A: 2, 3, 3, 4, 5. Group B: 1, 1, 2, 2, 3.
1. Why might a rank test be better than a t-test here? 2. Count the pairs: in how many of the 25 A versus B pairs does A give the higher rating? (Count a tie as half.)
Show the answer
1. Ratings are ordered categories, not measurements: the step from 1 to 2 needn't equal the step from 4 to 5. With only five people each, the t-test's assumptions are shaky. A rank test only uses the order.
2. A's 2 beats two 1s and ties two 2s: 2 + 1 = 3. Each 3 beats four and ties one: 4.5 each, 9 for the two. The 4 and the 5 beat all five: 10. Total 3 + 9 + 10 = 22 out of 25. A random A customer rates the menu higher than a random B customer about 88% of the time.
The words your lecturer uses
| Lecturer says | It means |
|---|---|
| Nonparametric test | A test that doesn't assume a particular shape, such as the bell curve, for the data |
| Rank | A value's position when all values are sorted |
| Mann-Whitney U test (Wilcoxon rank-sum) | The rank test for two independent groups |
| U statistic | The number of pairs one group wins (the smaller count is usually reported) |
| Wilcoxon signed-rank test | The rank test for paired data, such as before and after |
| Kruskal-Wallis test | The rank test for three or more groups |
| Robust | Not thrown off much by outliers or skew |
| Ties | Equal values, which share the average of their ranks |
Three things to remember
- Rank tests use only the order of values, so one extreme value can't swamp them.
- Mann-Whitney asks whether one group's values tend to beat the other's, not whether the averages differ.
- Use them for small, skewed or rating data. Use the average when the totals are what matter.
Part of Statistics From Zero, a free beginner course by WrongStat. Not a substitute for your lecturer, but hopefully a good translator.