The Algorithm Is 98.7% Accurate. So Is Saying No to Everyone.
The short version: when the thing you're looking for is rare, accuracy is almost meaningless. If 1.3% of pairs are compatible, a program that rejects everyone is right 98.7% of the time and finds zero couples. In our example, an app that suggests 200 matches, 100 of them good, is also 98.7% accurate. The questions that matter are: of the matches it suggests, how many are good (precision), and of the good matches out there, how many does it find (recall)?
Brenda, Aunt Linda's sister, is newly single and has joined a dating app. The app says its matching algorithm is 98.7% accurate. It was trained on millions of couples, it considers 400 personality factors, and its advert features a sunset that is 98.7% accurate.
In its first week, the algorithm matched Brenda with a man who described himself as "between hobbies", a man who described himself as "between wives", and a man who turned out to be her dentist. Brenda asked the app what the other 1.3% looks like.
The program that says no to everyone
Let's build a better algorithm. It's very short. For every pair of people, it says: "not compatible". That's the whole program.
Suppose that out of 10,000 possible pairs, 130 are genuinely compatible: 1.3%. Our program is wrong about those 130 and right about the other 9,870. That's 98.7% accurate. It matches the app's number to the decimal point, it took four seconds to write, and it has never introduced anyone to anyone.
This is why accuracy is a terrible number for rare things. When almost every answer is "no", saying "no" every time scores brilliantly. The base rate does all the work and the algorithm gets the credit.
What the app actually does
Now say the real app, out of the same 10,000 pairs, suggests 200 matches. 100 of them are good. It also misses 30 of the 130 good pairs.
- Right answers: 100 good matches plus 9,770 correctly rejected pairs = 9,870. That's 98.7% again.
- Of the 200 matches it suggests, half are good: 50%. This is called precision.
- Of the 130 good pairs out there, it finds 100: 77%. This is called recall.
The app and the "no to everyone" program have exactly the same accuracy. One of them finds 100 couples and the other finds none. Accuracy can't tell them apart. Precision and recall can.
Two programs, 10,000 pairs, 130 truly compatible
The two questions to ask
1. When it says yes, how often is it right? That's precision: the share of its suggestions that are any good. For Brenda, it's the share of dates worth the bus fare.
2. Of everything worth finding, how much does it find? That's recall: the share of good matches it actually catches. A cautious program with high precision can still miss most of them.
3. Compare with doing nothing. If "always say no" gets the same score, the score is measuring the base rate, not the algorithm.
Where it costs you money
"Our model predicted the market correctly on 99% of days" sounds impressive until you notice that crashes happen on about 1% of days, and a model that says "no crash" every morning scores the same. Any claim of high accuracy about a rare event (crashes, frauds, ten-baggers) needs its hit rate on the rare event itself: when the model says "crash", how often is it right, and how many crashes did it miss?
The moral
Brenda complained to the app. The app replied, with 98.7% accuracy, that her complaint had been received. She has since met someone in the queue at the post office, a method with a precision of one in one and a recall nobody has measured. The app sent her a survey asking whether its algorithm had helped. She said no. The app has recorded this as a correct prediction.
The app, its algorithm and all its numbers are fictional and illustrative. Educational content, not financial advice.