Stopping Early Without Fooling Yourself
You check your experiment every morning. On day 9 the effect crosses into significance and you stop. That habit — looking daily and stopping on the first good number — turns a 5% false-positive rate into something closer to 30%. Here's the fix, and it isn't 'stop looking.'
The Most Common Way Personal Experiments Go Wrong
You start a 20-day experiment. You have a dashboard. You look at it every morning, because of course you do — that's the point of tracking.
Day 6: nothing. Day 8: maybe something. Day 9: the effect crosses your threshold. Magnesium is working. You stop the experiment, declare a result, and add magnesium to your permanent routine.
This is the single most common way careful, well-designed personal experiments produce wrong answers. Not bad measurement. Not confounding. Just looking many times and stopping on the first look that says what you wanted to hear.
Why Repeated Looks Break Your Threshold
A significance threshold is a promise about a specific procedure: collect n observations, then test once. Under that procedure, a 5% threshold means a 5% chance of a false positive when nothing is going on.
Peeking breaks the procedure. Every additional look is another chance for random noise to wander across the line. You only need it to happen once, and you stop when it does.
Armitage, McPherson, and Rowe worked this out precisely in 1969. Testing at p < 0.05 with repeated looks at accumulating data:
| Number of looks | Actual false-positive rate |
|---|---|
| 1 | 5% |
| 2 | 8% |
| 5 | 14% |
| 10 | 19% |
| 20 | 25% |
And the limit is the part that should worry you: with unlimited looks, the probability of eventually crossing p < 0.05 on a true null effect is 1. Not high. One. Given enough looks at a null effect, you are guaranteed to see significance at some point, and if your rule is "stop when it's significant," you will always stop, and you will always stop on noise.
A daily-logged 20-day experiment that you check each morning is roughly the 20-look row. Your real false-positive rate is about 25% — five times what you think it is. Run four such experiments and, on average, one of them hands you a confident conclusion about nothing.
Bayesian Numbers Don't Rescue You
A common belief is that this is a frequentist problem, and that posterior probabilities are immune. Posteriors are better behaved — a posterior probability means what it says at every moment, regardless of when you look — but the stopping rule still does damage.
If your rule is "stop as soon as P(better) > 0.95," you are still selecting the moment the random walk happened to be at its highest. With an informative, well-calibrated prior, the prior pulls early extreme estimates back toward zero and this stays modest. With a flat or near-flat prior — which is what you have on your first experiment about a new intervention — early posteriors are volatile, and optional stopping inflates your error rate substantially. Not to 25%, but not to 5% either.
The general principle: no summary statistic saves you from a stopping rule that selects on that statistic.
Three Fixes That Actually Work
1. Pre-commit and don't stop (the boring one)
Decide the number of trials before you start. Look as often as you like — looking is free, stopping is what costs you. Run to the pre-committed end and evaluate once.
This is fully correct and requires zero mathematics. It is also the one people abandon, because watching a promising result and not acting on it is genuinely hard. If you know you won't hold the line, use option 2 or 3 instead of pretending.
2. Use a stricter early threshold
Group sequential designs solve exactly this problem in clinical trials, where stopping early can be an ethical obligation. The trick: make the bar for stopping much higher early, relaxing to normal at the planned end.
The O'Brien-Fleming approach (1979) is the well-known version. In practice, a serviceable approximation for a personal experiment:
- Before 50% of target trials: require P(better) > 0.99 to stop
- 50–80%: require P(better) > 0.97
- At the planned end: your normal threshold, 0.95
This preserves something close to your intended error rate while still letting a genuinely enormous effect end the experiment early. If the effect is large enough that you'd notice it without a dashboard, it will clear 0.99. If it isn't, you keep going — which is the correct answer.
3. Use always-valid inference
There are sequential methods designed so that the number is correct no matter how often you look or when you stop — mixture sequential probability ratio tests, confidence sequences, and related always-valid procedures. Johari, Koomen, Pekelis, and Walsh developed these for continuous monitoring of A/B tests, and they're what serious experimentation platforms use.
The cost is that always-valid intervals are wider than fixed-sample ones at any given moment. You trade precision for the freedom to stop whenever. For a personal experiment where you genuinely cannot resist acting on what you see, that trade is usually worth it.
Stopping for Futility Is Different
Everything above concerns stopping because something worked. Stopping because nothing is happening is a different situation with much gentler statistics.
If you're 70% through and the estimated effect is near zero with a tight interval, continuing is unlikely to change your decision. Abandoning is reasonable — the asymmetry is that you're not selecting an extreme result, you're declining to chase a flat one. The main cost is missing a small effect that would have emerged, and if the effect is that small, it probably wasn't going to change your behavior anyway.
The practical rule: be strict about stopping on success, relaxed about stopping on futility.
What This Means for How You Read a Running Result
The healthy way to watch an experiment in progress is as a forecast, not a verdict. An early separation is a hint about where things may land. It is not the answer, and its main use is deciding whether to keep going — not deciding what's true.
This is also why an evidence score should never be a dressed-up progress bar. Days logged tell you how disciplined you've been; they say nothing about whether the data has separated. A score driven mostly by how many days you've completed will read "strong" for an experiment that has learned nothing, and "weak" for one that has already found a decisive answer. What you want to see is the separation itself, with coverage as a secondary check on whether you've seen enough of your own variability to trust it.
Look every day. Let the number move you toward curiosity, not toward conclusions. Then stop when you said you would.
Try it: Before your next experiment starts, write the stopping rule into the hypothesis field — the target number of trials, and the early threshold you'd accept for stopping sooner. It takes ten seconds and it's the difference between a result you can trust and one you selected.
References
- Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society A, 132(2), 235–244.
- O'Brien, P. C., & Fleming, T. R. (1979). A multiple testing procedure for clinical trials. Biometrics, 35(3), 549–556.
- Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2), 191–199.
- Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. KDD '17.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology. Psychological Science, 22(11), 1359–1366.