How to Do A/B Testing in 2026 - A Practical Guide That Won't Lie to You

Most A/B testing goes wrong before the test even starts, and the mistake is almost always the same - someone ran a test, saw a green number, and shipped a change that was actually noise. A/B testing is not “show two versions and pick the winner.” It is a statistical procedure, and if you skip the statistics you get confident, wrong answers.

This is a practical guide to running A/B tests that tell you the truth. The process is the same whatever tool you use; I name specific platforms where they concretely fit, but the discipline is what protects you from false wins.

Step 1 - Write one hypothesis, pick one primary metric

Before anything else, write down what you believe and how you will know if you are right. A good hypothesis is specific - “moving the checkout button above the fold will increase completed purchases” - not “let’s try a new checkout.”

Then pick one primary metric. Just one. Completed purchases, signups, activation - whatever the change is meant to move. You can track secondary metrics, but you decide success on the primary one, chosen in advance. Picking your winning metric after you see the data is how people fool themselves.

Step 2 - Calculate sample size before you start

This is the step that separates real tests from theater. You cannot know if a result is trustworthy without knowing how much data you needed, and you have to work that out before the test runs.

A power calculation takes your baseline conversion rate, the smallest effect worth detecting, and your desired confidence, and tells you how many users per variant you need. If your calculation says you need 40,000 users per arm and you have traffic for 4,000, the test cannot give you a reliable answer no matter how green the number looks. Good platforms include a power calculator - GrowthBook, for instance, ships one on its Pro tier. Run the number first, or do not run the test.

Step 3 - Randomize at the user level

Split users into control and variant randomly, and do it at the user level, not the request level, so a single person always sees the same version. Inconsistent assignment - where a user flips between variants across page loads - poisons the result and confuses the user.

This is also where A/B testing meets feature flags. Under the hood, an A/B test is a flag that assigns each user to a variant, evaluated consistently through an SDK. That is why experimentation lives in the same tools as feature flags - the assignment mechanism is identical.

Step 4 - Run it, and do not peek

Start the test and let it reach the sample size from Step 2. Run it for at least one full business cycle - usually a week or two - so weekday and weekend behavior are both represented. A test that ran Tuesday to Thursday is measuring three days of one kind of user.

Here is the discipline that hurts - do not stop early because it looks significant. Repeatedly checking a running test and stopping the moment you see p < 0.05 dramatically inflates your false-positive rate. The result “reaches significance” and loses it constantly as data accumulates; catching it at a random high point is not a finding.

If you genuinely need to monitor continuously, use a tool built for it. Sequential testing is a statistical method designed to let you look at results as often as you like without corrupting them. Statsig and GrowthBook both include sequential testing in their stats engines - if you are the kind of team that will peek, use it rather than pretending you won’t.

Step 5 - Analyze honestly

When the test hits its sample size, analyze it - and analyze more than the headline.

• Statistical significance - is the difference real or noise? A proper engine computes this for you rather than leaving you to read p-values off a chart.
• Sample ratio mismatch (SRM) - did your 50/50 split actually land near 50/50? A skewed split signals a bug in assignment or logging, and it invalidates the result. Check it every time.
• Effect size - a statistically significant 0.1% lift may not be worth the engineering to ship. Significance is not importance.

Advanced engines add techniques that make tests faster and cleaner - CUPED reduces variance so you reach significance with less traffic, and Bayesian analysis gives you a probability the variant is better rather than a binary verdict. Statsig’s engine, built by ex-Facebook engineers, and GrowthBook’s warehouse-native engine both include CUPED, sequential testing, Bayesian analysis and SRM checks. These are not luxuries at scale - they are what stop you shipping noise.

Choosing a tool for the job

The right platform depends on your data setup and how central experimentation is.

• Statsig - all-in-one, and flag checks are free. It bundles flags, experimentation, analytics and session replay, with a stats engine that is the strongest in this category. Flag and config checks are unlimited and free on every tier; you pay on analytics events instead, with a flat $150/mo Pro tier. The free Developer tier gives 2M events with no card. One thing to price in - OpenAI acquired Statsig in September 2025, so a multi-year bet rides on a roadmap now inside OpenAI.

• GrowthBook - if you already have a data warehouse. It is warehouse-native, querying the BigQuery, Snowflake or Databricks warehouse you already run rather than ingesting a copy of your events, so your data never leaves your infrastructure. The MIT core self-hosts with unlimited users for free, and Cloud Pro is $40 per seat. The catch is real - it needs an instrumented warehouse and someone who understands the statistics, or you get a fraction of the value.

• Optimizely - if experimentation is your whole center of gravity. It popularized modern web experimentation and the stats engine is genuinely best-in-class, with a free Rollouts tier that includes unlimited flags but one experiment at a time. The gotcha is pricing - it publishes no paid price at all; every paid plan is a custom annual contract quoted on MAU and traffic. Third-party Vendr data puts the median Optimizely contract around $78,000/yr, but that covers the wider suite, not Feature Experimentation alone - treat it as directional.

The short version

A trustworthy A/B test is a sequence, not a single screen:

• One hypothesis, one primary metric, chosen before you start.
• Sample size calculated first - a test below its required power cannot answer you.
• Randomize per user, consistently.
• Run a full business cycle and do not peek - or use sequential testing if you will.
• Analyze for significance, SRM and effect size, not just the green number.

No tool saves you from skipping the process. The engine computes the statistics; the discipline is still yours.

---

Tool facts verified against each vendor’s pricing on 26 July 2026 via our experimentation tool reviews. Contract figures for Optimizely are from Vendr’s third-party buyer data and attributed as such. Pricing changes often - we re-verify regularly.