Set a Fixed Analysis Horizon Before an A/B Test Starts

Stakeholders often ask for early results while an A/B test is still running. Under business pressure the request is understandable, but acting on an early look, even once, can invalidate the experiment.

The statistics are unforgiving. Checking results repeatedly and stopping as soon as significance appears inflates the Type I error rate. An intended 5% false-positive rate can rise to 15%, 20% or more, depending on how often the data is checked. Simulations show the mechanism clearly: every extra look gives random noise another chance to cross the threshold, so an early 'win' says little about the treatment.

The remedy is simple. Commit to a fixed analysis horizon before launch. Run a power analysis to set the sample size and duration required for the minimum detectable effect, and do not evaluate the primary metric until that point is reached. The exercise also forces the team to state the hypothesis precisely and agree on what size of effect matters.

Sometimes stopping early is genuinely necessary. Sequential designs, such as group-sequential tests with O'Brien-Fleming boundaries, allow it without inflating the error rate, but they add complexity and are often implemented incorrectly in practice. For most teams, a fixed horizon combined with clearly defined, version-controlled metrics, for example in dbt, is the more dependable choice. It also curbs the temptation to over-read noisy early data.

A disciplined horizon cannot rescue poor inputs. If the event data feeding the test is incomplete or biased, a perfectly timed analysis still produces questionable results. Reliable experimentation starts with how the data is collected and ends with a disciplined analysis.