Are Performance Reviews Accurate? What a Rating Really Measures
The short version: a single performance rating tells you more about the rater than about the person rated. In the best-known study, 4,492 managers were each rated by two bosses, two peers and two subordinates; about 62% of the variation in ratings was down to the individual rater's quirks and about 21% to actual performance. In our simulation of a team of ten, the person rated best is truly the best only 27% of the time, comes from the bottom half 22% of the time, and two independent raters pick the same star in only about one team in six.
Review season. Kevin got a 3 out of 5: "meets expectations". Darren, who sits next to him, got a 4. Kevin knows exactly what Darren does all day, and it's mostly forwarding Kevin's emails with "see below".
Darren's manager is generous and gives everyone 4s. Kevin's manager believes that 5 is for God and 4 is for people who have saved a life. Both managers think their ratings are accurate.
What a rating is made of
A rating mixes several things: how well the person actually performs, how the rater sees the world (strict or generous, what they notice, whom they like), the rater's position (a boss sees different things from a colleague), and plain random noise. The question is how big each part is.
The best-known answer comes from a 2000 study of 4,492 managers. Each was rated on the same set of skills by two bosses, two peers and two subordinates, which let the researchers separate what all the raters agreed on from what was particular to each one.
| Part of the rating | Share of the differences in ratings |
|---|---|
| The individual rater's quirks | about 62% |
| The person's actual performance | about 21% |
| The rater's position and random noise | the rest |
Put simply, when two people get different ratings, the most likely reason is that they were rated by different people. A rating measures the rater about three times as much as it measures the person rated.
What a performance rating is made of
What that does to a team of ten
We simulated 400,000 teams of ten, where each rating is 21% real performance and the rest noise, and the person with the top rating gets the "exceeds expectations".
| Outcome | With ratings like these | Pure chance |
|---|---|---|
| Top-rated person is truly the best | 27% | 10% |
| Top-rated person is from the bottom half | 22% | 50% |
| Two independent raters pick the same star | 17% | 10% |
Ratings are clearly better than drawing names from a hat, and clearly worse than what the forms suggest when they ask for a number to one decimal place.
Statistics for noobs: measurement error
Every measurement is the true value plus some error. A bathroom scale that adds a random kilo is still useful over a month, useless for deciding whether you lost 200 grams since yesterday. Measurement error is the reason a single noisy number shouldn't decide anything important.
The usual fixes are the same everywhere: measure more than once, use more than one rater, average, and make the scale concrete ("closed 30 tickets with no reopening") rather than a feeling ("strong communicator").
Where it costs you money
Fund ratings, analyst recommendations and "star" ratings are measurements with their own errors and their own raters' habits. One analyst's "strong buy" partly measures the analyst. Several independent opinions, or better, a long, comparable track record, tell you more about the thing being rated.
The moral
Kevin's company decided to fix the problem by making all managers calibrate their ratings in a long meeting. In the meeting, the manager who talked loudest got his team's ratings raised. The company now has a second layer of rater effect on top of the first. Darren has been rated 4 by both of them, which, the system says, proves the rating is reliable.
What this can't tell you
- One kind of rating. The study used ratings of managers' skills from several sources, collected for development, not pay. Ratings tied to bonuses may be even more distorted, or less.
- Our simulation is a model. It treats everything except performance as noise that differs between raters; in reality some biases are shared, for example by everyone in the same office.
- Concrete measures help. Jobs with clear outputs (sales, units produced) can be rated much more accurately than jobs judged on impressions.
Kevin, Darren and their managers are an illustration. The 62% and 21% figures come from a 2000 study in the Journal of Applied Psychology, as widely cited, including in the Harvard Business Review in 2015. The team-of-ten percentages come from our own simulation of 400,000 teams. Educational content, not career, legal or financial advice.