How to Run an A/B Test in 2026 - A Step-by-Step Practical Guide

Most A/B tests fail before they start. Not because the tooling breaks, but because the test was designed to produce a confident answer whether or not there is a real effect. The hard part of A/B testing is not running the test - it is running one you can actually trust. Peeking at results, stopping early, testing without a sample-size target, and calling a noisy 1% lift a win are how teams ship changes that do nothing and congratulate themselves for it.

This is a practical guide to running an A/B test that gives you a real answer. The process matters more than the tool, but I name tools where they concretely fit.

Step 1 - Write a hypothesis, not a hunch

Before anything, write down the change, the metric it should move, and the direction. “Changing the CTA from blue to green will increase signup conversion” is a hypothesis. “Let’s try green” is not.

The discipline here is picking one primary metric up front. If you decide after the fact which metric to report, you will always find one that moved, because with enough metrics something always does by chance. Pick the primary metric before launch and hold the rest as secondary context, not as backup winners.

Step 2 - Calculate your sample size before you launch

This is the step teams skip, and it is the one that decides whether the test can conclude anything. Your required sample size depends on two things: your baseline conversion rate, and the smallest lift worth detecting. A smaller effect needs far more users to prove - detecting a 1% relative lift can take an order of magnitude more traffic than a 10% one.

Run a power calculation before you start and get a real number. Then commit to running until you hit it. GrowthBook includes a power calculator on its Pro tier, and any serious platform will help you size the test. The failure mode is running to a deadline instead of a sample size, which leaves you underpowered and unable to tell a real effect from noise.

Step 3 - Assign variants consistently

Each user must see the same variant every time, or the test is meaningless. That means hashing a stable identifier - a user ID or a persistent session ID - to bucket users deterministically, not flipping a coin on each page load.

This is where a real tool earns its keep. A/B testing runs on the same infrastructure as feature flags: you are just serving different variants to different buckets and measuring the outcome. Statsig was built by ex-Facebook engineers specifically to mirror a rigorous internal experimentation stack, and consistent assignment is table stakes for it. If you roll your own bucketing, inconsistent assignment is the first thing that quietly corrupts your data.

Step 4 - Instrument the metric properly

Your test is only as good as the events behind it. Make sure the primary metric is tracked cleanly for both variants, fired once per event, and attributed to the right bucket. A metric that double-fires in one variant will invent a lift that does not exist.

GrowthBook takes a distinctive approach here: it is warehouse-native, so it queries the BigQuery, Snowflake or Databricks warehouse you already run rather than ingesting a separate copy of your events. Your analytics data never leaves your infrastructure. The trade is that you need a properly instrumented warehouse already - no warehouse, far less value. If your event data already lives in a warehouse, that is a clean way to run experiments on the source of truth.

Step 5 - Run to completion, and stop peeking

Launch, then leave it alone. Run until you hit the sample size from Step 2 and at least one full business cycle - a week or two - so weekday and weekend behaviour are both represented.

Stopping the moment a result looks significant is the single most common way to ship a false winner. Early significance is frequently noise that regresses to nothing with more data. If you genuinely need the option to stop early, use an engine built for it: sequential testing corrects the statistics for repeated looks. Statsig and GrowthBook both include sequential testing for exactly this reason. Without it, decide your stopping rule up front and hold to it.

Step 6 - Read the result honestly

When the test concludes, check three things, not one:

1. Statistical significance - is the difference unlikely to be chance? A result significant at 95% means roughly a 5% chance of seeing that difference if the variants were truly identical. Our guide on statistical significance in A/B testing goes deeper.
2. Practical significance - is the effect big enough to matter? A statistically significant 0.1% lift may not be worth the code it takes to ship.
3. Sample ratio - did roughly the traffic split you intended actually land in each bucket? A sample-ratio mismatch means something is broken in assignment and the result is untrustworthy. GrowthBook runs SRM checks automatically.

A proven stats engine does the significance maths for you. Optimizely popularised modern web experimentation and its stats engine decides significance so you are not eyeballing p-values - though its paid pricing is entirely sales-led, with no published number, so start on its free Rollouts tier if you want to try it.

Step 7 - Ship or roll back, then document

Act on the result. If the variant won on your primary metric with adequate power and a meaningful effect, ship it. If it lost or showed no effect, roll it back - a null result is a real result, and knowing a change does nothing saves you from shipping it everywhere.

Either way, write down the hypothesis, the numbers, and the decision. A test you cannot find in six months will get re-run by someone who did not know it was already answered.

The short version

• Design first: one hypothesis, one primary metric, a calculated sample size - before launch.
• Assign consistently off a stable identifier, and instrument the metric cleanly.
• Run to completion and stop peeking, unless you use sequential testing built to allow early looks.
• Read three things: significance, practical effect size, and sample ratio.
• Tools: Statsig for a rigorous engine with free flag checks, GrowthBook if your data is warehouse-native, Optimizely for a proven stats engine on a free on-ramp.

A trustworthy A/B test is mostly discipline you apply before the test runs, not cleverness after. For the wider playbook see how to do A/B testing, and to compare engines head to head, the best A/B testing tools.