How to Design an A/B Test in 2026 - A Complete Framework

Almost every bad A/B test was doomed before it launched. Not by a bug, but by a design that guaranteed a confident answer whether or not a real effect existed. The teams who trust their results are not cleverer at analysis - they are more disciplined at design. A trustworthy A/B test is won before a single user is bucketed, in the decisions you lock down while the experiment is still on paper.

This is the design framework, step by step. It is deliberately separate from actually executing a test - for that, see how to run an A/B test. Here we stay entirely in the planning stage, because that is where the trust comes from.

Step 1 - Write a falsifiable hypothesis

Start with a sentence that names the change, the metric it should move, and the direction. “Moving the pricing link into the main nav will increase trial signups” is a hypothesis. “Let us try moving the pricing link” is a wish.

A good hypothesis is falsifiable - it makes a prediction that the data can prove wrong. If you cannot state in advance what result would make you abandon the idea, you are not designing a test, you are looking for validation. Write down the mechanism too - why you expect it to work - because that reasoning is what you will interrogate if the result surprises you.

Step 2 - Choose one primary metric

Pick the single metric that decides the test, before launch. This is your Overall Evaluation Criterion, and committing to it is the most important anti-cheating rule in experimentation.

The reason is blunt: track ten metrics and, with a 5 percent false-positive rate each, you have a very good chance that at least one moves “significantly” by pure luck. If you choose which metric to report after seeing the data, you will always find a winner. Fixing one primary metric up front is what stops you from mining noise for a story. Everything else you track becomes secondary context, not a backup source of victories.

Choose a metric that is sensitive enough to move within the test, and close enough to real value that moving it actually matters. Optimising a vanity metric nobody cares about is a design failure, not an analysis one.

Step 3 - Define your guardrail metrics

Now name the things the change should not break. Guardrail metrics are measures you do not expect to improve but refuse to let regress - latency, error rate, crash rate, refund rate, unsubscribes, support tickets.

A variant that wins on the primary metric but quietly degrades a guardrail should be blocked, not shipped. A checkout redesign that lifts conversion 3 percent while adding 400ms of latency and 1 percent more errors is not a win - it is a trade you never agreed to. Defining guardrails at design time is what turns that from a surprise into a rule. The guardrail metric glossary entry has the short version.

Step 4 - Set the minimum detectable effect

The minimum detectable effect (MDE) is the smallest lift you want the test to catch reliably. It is a design decision, and it is the single biggest lever on how much traffic you will need.

Set it to the smallest change that would actually be worth shipping, not the smallest number you can imagine. If a 0.2 percent lift would not change your decision, do not design a test powered to detect 0.2 percent - it would demand enormous traffic to resolve a difference you would ignore. A realistic MDE keeps the test affordable. An over-ambitious one makes it impossible.

Step 5 - Calculate the sample size and duration

With a baseline rate, an MDE, a significance level (usually 5 percent) and a power target (usually 80 percent), a power calculation gives you the sample size you need. This is not optional maths - it is the number that decides whether your test can conclude anything at all.

Then convert sample size to duration using your traffic. Two rules override a naive division:

• Run at least one full business cycle, usually a week or two, so weekday and weekend behaviour are both represented. A test that only saw Tuesdays has not seen your users.
• Do not run to a deadline instead of a sample. A test stopped at an arbitrary date because the sprint ended is underpowered by accident.

GrowthBook includes a power calculator on its Pro tier, and any serious platform will size the test for you.

Step 6 - Fix the randomisation unit and assignment

Decide what gets randomised - usually the user, sometimes the session, occasionally a cluster like an account or a geography. This matters because your unit of analysis must match your unit of randomisation, or your significance maths is wrong.

Assignment must be deterministic: hash a stable identifier so a user always lands in the same bucket. Statsig, built by ex-Facebook engineers to mirror a rigorous internal experimentation stack, treats consistent assignment as table stakes, and its flag and config checks are unlimited and free on every tier. Inconsistent bucketing is a leading cause of sample ratio mismatch, the broken-split signal that invalidates a test outright.

Step 7 - Write the analysis plan before you launch

Finally, decide now how you will read the result - while you still cannot be tempted by the data. Your plan should state:

• The stopping rule: a fixed sample size with no peeking, or a sequential-testing engine that allows early looks. Pick one and never mix them.
• The exact test and significance threshold you will apply to the primary metric.
• Which segments you will look at, declared in advance so post-hoc slicing does not become fishing.
• The SRM check you will run before trusting anything.

An analysis plan written before launch is what separates a real experiment from a story assembled after the fact. Optimizely built its Stats Engine around always-valid inference so continuous monitoring does not break the maths, with a free Rollouts tier and sales-led pricing above it.

The design checklist

• Hypothesis: falsifiable, with a stated mechanism.
• Primary metric: exactly one, chosen before launch.
• Guardrails: what the change must not harm.
• MDE: the smallest lift worth shipping.
• Sample size and duration: power-calculated, one full business cycle minimum.
• Randomisation unit: fixed, deterministic assignment.
• Analysis plan: stopping rule, test, segments and SRM check, all written before launch.

Get these seven right and the run is almost boring - you execute the plan and read the number. For the execution side see how to run an A/B test, for what the result means see the statistical significance guide, and to pick the engine that carries the stats, the best experimentation platforms roundup. The test you design carefully is the one you can actually trust.