Optional stopping
Deciding when to end an experiment based on the results so far, for example stopping as soon as they look significant. It makes false positives much more likely.
In plain English
A significance test's 5% false alarm rate assumes you look once, at a point chosen in advance. Every extra look is another chance for noise to cross the line, and early results, based on little data, jump about the most. The fixes are to fix the sample size before starting, to use a method built for repeated looks with a stricter threshold at each one, or to confirm the result on fresh data.
Uncle Gary explains
Marketing checked a test of two identical buttons every morning. In our simulation, daily checking for 20 days declared a false winner 25% of the time, against 5% for one look at the end.
Why it matters for your money
Checking a new strategy daily and going all in the first good week, or ending a backtest on its best date, is the same mistake.
Where we tested it
Related terms
All glossary terms. Educational content, not financial advice.