Corporate life · Murphy's laws
Sturgeon's Law: 90% of Ideas Are Bad. What Does That Do to the Ideas Box?
The short version: when most candidates are bad, even a decent filter passes mostly bad ones. If 10% of ideas are good and a panel judges correctly 80% of the time, it approves 26% of ideas, and only 31% of those are good. That's a base-rate effect: the 20% of bad ideas it wrongly approves come from a much bigger pile. A second independent round lifts the good share to 64%, a third to 88%, at the price of losing more good ideas each time (36% after two rounds). The share of approved ideas that are really good is the filter's precision.
The innovation programme received 200 ideas this year. A panel of five senior managers rated them, and 52 were approved for funding. The CEO announced "52 game-changing ideas".
A year later, about 16 look promising. Kevin's boss blames the panel. The panel blames the ideas. Kevin blames a science-fiction writer from the 1950s.
Sturgeon's law
Theodore Sturgeon, defending science fiction against critics who said most of it was rubbish, replied that 90% of everything is rubbish: novels, films, consumer goods. Sturgeon's law has since been applied to emails, meetings and strategy documents. Let's apply it to the ideas box, and assume, generously, that only 90% of ideas are bad.
A good panel on a weak pool
Suppose the panel is quite good: it approves 80% of good ideas and rejects 80% of bad ones. Out of 200 ideas:
| Good ideas (20) | Bad ideas (180) | |
|---|---|---|
| Approved | 16 | 36 |
| Rejected | 4 | 144 |
The panel approves 52 ideas, and only 16 of them, 31%, are good. It wrongly approves just 20% of bad ideas, but 20% of a big pile is bigger than 80% of a small one. That's the base rate at work: when the thing you're looking for is rare, most of what a filter catches is the thing you're not.
The share of approved ideas that are really good is the filter's precision. A panel can be "80% accurate" and still have a precision of 31%.
A second round
Run the approved ideas past a second, independent panel just as good as the first, and then a third:
| Rounds of judging | Ideas approved | Of those, good | Good ideas lost on the way |
|---|---|---|---|
| 1 | 26% | 31% | 20% |
| 2 | 10% | 64% | 36% |
| 3 | 6% | 88% | 49% |
Each round cleans the pile, and each round also throws away some good ideas. The trade-off is unavoidable; the only question is which mistake costs more: funding duds, or losing gems. A cheap first round followed by a small real test of each survivor often beats a single big judgement.
Share of approved ideas that are really good, by rounds of judging
Statistics for noobs: precision depends on the pool
"Our screening is 80% accurate" says nothing about how many of its picks are good until you know how common good candidates are. The rarer the target, the lower the precision, however careful the judges. It's the same arithmetic as medical screening and fraud alerts.
Where it costs you money
If only a small share of stocks, funds or startups go on to be big winners, a stock-picker who is right most of the time still ends up with a portfolio full of non-winners. Before trusting someone's hit rate, ask how rare the thing they're picking really is.
The moral
Kevin suggested a second round of judging. The CEO liked it, and asked the same five managers to judge the same 52 ideas again, "to be thorough". They approved all 52, because they'd approved them the first time. Kevin has noted that a second round only works if it's independent. The programme now has a third round, which the five managers have also passed.
What this model can't tell you
- The numbers are assumptions. Real pools may have more or fewer good ideas, and panels may be better or worse than 80%.
- "Good" isn't always clear. Many ideas are neither good nor bad until someone tries them; a small test often tells you more than a panel.
- Rounds aren't always independent. The same judges, or judges who talk to each other, add much less than a fresh panel.
The innovation programme, the panel and the CEO are an illustration. The figures are our own calculation, assuming 10% of ideas are good and each independent round of judging is right 80% of the time. Educational content, not business or financial advice.