Statistical Significance

What it means

Statistical significance asks a simple question: if the variant truly had no effect, how often would you see a difference this large by chance alone? That probability is the p-value. A common rule declares a result significant when the p-value falls below 0.05, an under-5-percent chance of a false alarm. It is paired with statistical power, the chance of detecting a real effect if one exists, which is what your sample size buys you. Significance tells you the result is unlikely to be noise; it does not tell you the effect is large or worth shipping.

Why it matters

Calling a winner too early is the most common experimentation mistake. Metrics swing wildly at low sample sizes, so a variant can look like a runaway success on day one and settle to break-even by day ten. A significance threshold, fixed before the test starts, is the guard against reading noise as signal and shipping changes that do nothing.

In practice

Good platforms compute significance for you so you are not eyeballing p-values. Optimizely’s stats engine decides significance rather than leaving it to interpretation, and GrowthBook and Statsig expose full statistical engines. Two techniques make significance faster and safer to reach: CUPED reduces variance so you need less traffic, and sequential testing lets you peek at results without inflating the false-positive rate. See statistical significance in A/B testing for the practical rules.