What does 'no evidence of a problem' mean?

Only that a check didn't find the problem. Whether that means anything depends on how good the check was. On a road where 5% of stretches have a pothole, inspecting two stretches finds nothing about 90% of the time, even though the potholes are there.

What is the difference between absence of evidence and evidence of absence?

Absence of evidence describes the search: nothing turned up. Evidence of absence describes the world: the thing isn't there. You can only move from one to the other if the search was big and careful enough to find the thing if it existed.

What is statistical power?

The chance that a test or check finds an effect or problem that really exists. Small checks have low power, so 'we found nothing' from a small check is weak evidence that nothing is there.

Real life · Statistical power

The Committee Found No Evidence of Potholes. It Checked 20 Yards.

About 5 minutes · No trading required

The short version: finding nothing only means something if your check could have found it. On a 2,000-yard road with 10 potholes, inspecting 20 yards finds none about 90% of the time. To have a 95% chance of spotting at least one, you need to check about 590 yards. 'No evidence of a problem' from a small check is not evidence of no problem. Statisticians call the ability of a check to find something that's really there its power.

Uncle Gary sits on the Village Road Safety Committee. The committee was formed after complaints about potholes on Mill Lane, and its first act was to form a subcommittee to decide whether the complaints were within the committee's remit. They were. This took four months.

The inspection took place on a Tuesday evening. It was raining, so the committee inspected the 20 yards of Mill Lane directly outside the village hall, where the biscuits were. The committee's torch was switched off halfway through, because it was shining in the chairman's eyes.

The report was unanimous: no evidence of potholes. It was published under the title "Evidence-Based Success". On the way home, the chairman's bicycle was eaten by a pothole 40 yards further down Mill Lane. The committee has scheduled a meeting to discuss whether the bicycle was within its remit.

How likely was the committee to find anything?

Let's give Mill Lane some numbers. It's 2,000 yards long, which we'll split into 200 stretches of 10 yards. Ten of those stretches have a pothole: 5% of the road. The committee looked at two stretches.

The chance that a single stretch has no pothole: 95%.
The chance that both stretches the committee checked have none: about 90%.

So with ten potholes on the road, the committee's inspection would come back clean nine times out of ten. Its report says nothing about Mill Lane. It says a lot about the inspection.

Chance of finding at least one pothole, on a road with ten

20 yards10%

100 yards40%

300 yards79%

590 yards95%

A 2,000-yard road split into 200 stretches, 10 of them with a pothole; each stretch treated as an independent draw.

No evidence of a problem is not evidence of no problem

There are two different sentences here, and committees love to swap them. "We found no potholes" describes the check. "There are no potholes" describes the road. You only get from the first to the second if the check was good enough to find potholes that were there.

The ability of a check to find something that really exists is called statistical power. A tiny inspection, a short trial or a small survey has low power: it will usually find nothing, whether or not something is there. Finding nothing with a low-power check is like searching for your keys with your eyes shut and concluding that you don't own any keys.

How much road do you have to check?

Still assuming 10 potholes on the road:

Check 100 yards: you find at least one about 40% of the time.
Check 300 yards: about 79%.
Check about 590 yards: about 95%. Only now does "we found none" start to mean "there probably aren't ten".

And fewer potholes need a bigger check. Rare problems hide well. That's why one person in a clinical trial, one week of a new routine or one quiet street tells you almost nothing about whether something is safe, effective or fixed.

How to read "we found no evidence"

1. Ask how big the check was. 20 yards, 12 patients, three weeks. If the answer is small, the "no evidence" is small too.

2. Ask what the check could have found. Would a real problem of a realistic size have shown up? If not, finding nothing was the expected result.

3. Ask who switched off the torch. A check that was cut short, done in the dry bit or done where the biscuits were has been made weaker, whether anyone meant to or not.

Where it costs you money

"We found no evidence that this strategy loses money", after two months and nine trades, is the committee's report on Mill Lane. A test that small would usually find nothing either way. Before you trust a clean result, ask whether the test was big enough to catch a real loss. That's why the tests on this site use years of data and thousands of trades, and why we say so when a sample is too small to tell.

The moral

The committee met again to consider the bicycle. It voted to commission a full inspection of Mill Lane, on the condition that it be carried out by someone who is not on the committee, in daylight, with the torch on. Nobody could be found who met all three conditions, so the inspection has been postponed. The potholes have formed a residents' association.

Mill Lane, its committee and its potholes are illustrative; the calculations treat each stretch of road as an independent draw. Educational content, not financial advice.

More statistics from real life