Sequential Testing and the Peeking Problem in A/B Tests (2026)

There is a mistake so natural that almost every team makes it, feels responsible doing it, and quietly wrecks their results in the process. It is watching an A/B test tick upward, seeing it cross the significance line, and calling the winner right then. Stopping a test the moment it looks significant is the single most common way to ship a false winner, and the statistics behind why are worth understanding before you trust another result.

Why peeking breaks the math

A standard significance test - the fixed-horizon test most people learned - comes with fine print: you look once, at a sample size you committed to in advance. The 5 percent false-positive rate it promises is calculated for exactly one look.

The trouble is that an in-progress test does not sit still. As data trickles in, the measured difference wanders up and down around its true value. Early on, with little data, it wanders a lot. If you check repeatedly and stop the first time the line is crossed, you are not measuring the effect - you are cherry-picking the noisiest, most favourable moment.

Every extra look is another independent chance for random noise to cross the threshold, so the more you peek, the higher your real false-positive rate climbs. Check a handful of times and stop opportunistically and a test built for 5 percent error can easily run at 20 or 30 percent. Check continuously and, in the limit, a null test will eventually cross the line essentially every time - because with enough looks, pure noise always wanders far enough sooner or later. You would declare winners for changes that do nothing, over and over, and never know it.

The two honest ways out

There are exactly two disciplined responses, and most teams only know the first.

Option one - fix the sample size and do not look. Run a power calculation before you start, commit to the resulting sample size, and refuse to make a decision until you hit it. This is the classic, correct approach for a fixed-horizon test. It works, but it is inflexible: you cannot stop early even when the effect is huge and obvious, and you cannot react in real time.

Option two - use sequential testing. This is a class of statistical methods purpose-built to let you look as often as you like. The idea is to spread your error budget across the looks rather than spend it all at one point, so the threshold you compare against adjusts to account for continuous monitoring. With sequential stats, early stopping is not cheating - it is the entire point.

How sequential methods actually work

A few distinct flavours show up in real platforms, and it helps to know which is which.

• Group sequential designs. You pre-define a small number of interim analyses - say, four check-ins - and use a spending function such as O’Brien-Fleming or Pocock that makes the early thresholds stricter and loosens them as you go. You get a few honest chances to stop early.
• Sequential probability ratio test (SPRT). A running likelihood ratio that accumulates evidence for each hypothesis with every observation, letting you stop as soon as the evidence passes an upper or lower boundary.
• Always-valid p-values. A modern construction, based on mixture sequential probability ratio tests, that produces a p-value valid at every point in the test simultaneously. You can look continuously, in real time, and stop whenever it crosses your threshold. This is the approach Optimizely popularised in its Stats Engine.

The common thread is that all of them trade a little peak power for the freedom to look whenever you want. You give up a small amount of sensitivity in exchange for the ability to monitor a live test and still trust the answer.

Which platforms give you this

The engine is the feature here, so platform choice matters.

Optimizely built its Stats Engine around always-valid inference precisely to solve this problem, so results are designed to hold up under the continuous watching that breaks naive tests. It has a free Rollouts tier, though its paid experimentation pricing is entirely sales-led with no published number.

Statsig includes sequential testing in the engine its ex-Facebook founders built to mirror a rigorous internal experimentation stack, alongside CUPED variance reduction. Flag and config checks are unlimited and free on every tier, with analytics events as the meter. It has been owned by OpenAI since September 2025.

GrowthBook carries sequential testing too, plus CUPED, Bayesian analysis and SRM checks, and being warehouse-native it runs them against your own BigQuery, Snowflake or Databricks. The MIT core self-hosts free, with the usual trade of a real statistics learning curve.

The rule to actually remember

Boil it down to one sentence and it is easy to hold onto. Fixed sample size means no peeking; sequential stats means peek all you like - pick one and never mix them. The failure mode is running a plain fixed-horizon test and then watching it like a stock ticker, which gives you all the temptation of early stopping and none of the statistical protection.

For the surrounding practice, our statistical significance guide unpacks what a p-value really claims, how to run an A/B test builds the stopping rule into a full workflow, the sequential testing glossary entry gives the one-line definition, and the best experimentation platforms roundup shows which engines actually implement it. The discipline is cheap; the false winners you avoid are not.