Sequential Testing

What it means

In a classic fixed-horizon test you must pick a sample size up front and only check significance once, at the end. Peek early and repeatedly at a 0.05 threshold and your true false-positive rate balloons, because each look is another chance to catch a random swing. Sequential testing fixes this with always-valid inference: the significance boundary widens to account for continuous monitoring, so you can watch the results every day and stop the moment they cross the line, whenever that happens, while keeping the error rate you signed up for.

Why it matters

Fixed-horizon tests waste time and traffic. If a variant is clearly winning or clearly harmful after three days, waiting two more weeks to hit a pre-set sample is pure cost, and in the harmful case it is user damage. Sequential testing lets teams ship winners sooner and kill losers faster while keeping statistical significance honest. The price is a modest increase in the sample needed for a borderline result.

In practice

Sequential testing signals a real stats engine. GrowthBook lists sequential testing next to CUPED, Bayesian analysis and SRM checks, and Statsig was built to mirror Facebook’s internal stack with sequential testing at its core. Both let product teams make valid early calls on an A/B test instead of eyeballing a moving chart. See statistical significance in A/B testing for why continuous peeking otherwise breaks your results.