Corporate life · Murphy's laws
Twyman's Law: Any Number That Looks Interesting Is Probably Wrong
The short version: when a number looks spectacular, the most likely explanation is an error, because errors are common and spectacular real changes are rare. If a genuine jump of 50% or more in conversions happens on about 1 day in 1,000, and a tracking glitch that produces such a jump on about 1 day in 100, then out of 11 spikes, 10 are glitches: a spike is real about 9% of the time. That's Twyman's law, popular among people who run online experiments. Check the plumbing before writing the press release.
Tuesday morning. The marketing dashboard shows website conversions up 400% overnight. The new banner went live yesterday. By 9:30 there's a draft press release, a cake order and an email to the CEO titled "Game changer".
Kevin opens the raw data. Every conversion since midnight has the same timestamp to the millisecond, and they all came from one IP address in the company's own office. Darren had been testing the checkout page. Repeatedly.
The law
Tony Twyman, a British researcher who analysed TV and media audiences, gave his name to a short rule: any figure that looks interesting or different is usually wrong. It has become a favourite of people who run online experiments, who see more spectacular, wrong numbers than anyone.
It sounds cynical. It's actually a statement about base rates.
Why the exciting number is usually the wrong one
Two things can make a dashboard spike: something really changed, or something broke. Suppose, for a typical website, a genuine jump of 50% or more happens on about 1 day in 1,000, and a tracking glitch that produces such a jump (a duplicated tag, a bot, a test account, a currency mix-up) on about 1 day in 100. Over 1,000 days:
| Cause of a big spike | Days out of 1,000 |
|---|---|
| Something really changed | 1 |
| Something broke in the data | 10 |
So when a spike appears, it's real about 1 time in 11, about 9%. The figures are our assumptions, but the shape holds whenever errors are more common than miracles, which is almost everywhere.
The same logic works in reverse: a sudden, catastrophic drop is also more likely a broken tracker than a collapse in demand. Twyman's law is even-handed. It distrusts good news and bad news equally, as long as they're dramatic.
The five-minute checklist
- Look at the raw rows. Duplicates, identical timestamps, one source or one IP address.
- Check a second source. Do orders, payments or server logs show the same jump?
- Look for a change in measurement. New tag, new page, new filter, new definition, new time zone.
- Ask if it's plausible. Did anything happen that could cause a change this big? A banner rarely quadruples sales.
Statistics for noobs: extreme values earn extra checks
An outlier is either the most interesting point in the data or a mistake, and mistakes are more common. Before building a story on an extreme value, ask how often errors of that size happen compared with real changes of that size. The more surprising the number, the more evidence it needs.
Where it costs you money
A backtest showing 200% a year with no losing months, or a stock screener flagging a company at a price-to-earnings ratio of 0.3, is far more likely to contain a data error than a hidden fortune. Spectacular results need spectacular checking.
The moral
The press release was cancelled. The cake had already arrived. Marketing decided to eat it anyway, in celebration of the new banner, which, once Darren's test orders were removed, had raised conversions by 2%. Darren has been asked to test the checkout from his phone in future. The dashboard now shows a 400% rise in mobile conversions.
What this can't tell you
- Our rates are assumptions. Some data pipelines are very clean, some very messy; the 9% changes with them.
- Sometimes it is real. A viral mention or a big new customer can produce a genuine spike. The law says check first, not dismiss.
- The law's wording varies. It's quoted in several forms; the idea is the same.
The dashboard, the cake and Darren's test orders are an illustration. Twyman's law is widely quoted, including in a 2020 book on online experiments by Kohavi, Tang and Xu. The 1-in-11 figure is our own calculation from assumed rates. Educational content, not business or financial advice.