A/B Test

What it means

An A/B test randomly divides traffic into two arms. The control group A sees the current experience; the treatment group B sees one change. You pick a primary metric ahead of time, run both arms in parallel, and compare the results. Because assignment is random, the two groups are similar in every way except the change, so any reliable difference in the metric can be credited to the variant rather than to seasonality, marketing, or luck.

Why it matters

Opinions about what users want are cheap and usually wrong. An A/B test replaces the argument with evidence. It also protects you from shipping changes that feel like improvements but quietly hurt conversion, retention, or revenue. Running the test on a rollout percentage means you can hold the experiment at a small slice of traffic and stop early if the variant is clearly worse.

In practice

Feature flag platforms increasingly ship real experimentation engines rather than a chart bolted onto a toggle. GrowthBook is warehouse-native and runs the actual statistics, and Statsig was built by ex-Facebook engineers to mirror an internal experimentation stack. The hard parts are not the split but the analysis: you need enough sample to reach statistical significance, and you need to watch for a sample ratio mismatch that signals the split itself is broken. See how to run an A/B test for a full walkthrough.