The p-value in A/B Testing, Explained in Plain English (2026)

The p-value is the most reported and most misunderstood number in A/B testing. Almost every tool prints it, almost every team treats 0.05 as a finish line, and a large fraction read it backwards. Getting it wrong does not just make you sound sloppy - it makes you ship changes that do nothing and kill changes that would have worked. This is the plain-English version.

A p-value is the probability of getting a result at least as extreme as yours, assuming the two variants are actually identical. Read that twice, because the “assuming the variants are identical” part is the whole game and it is exactly the part people drop.

What the number is really measuring

Imagine A and B are truly the same - same green and blue button, no real difference in behavior. Even then, random chance means one will pull ahead in any finite sample, just like flipping two fair coins 500 times each rarely gives an exact tie.

The p-value answers - in that no-difference world, how often would luck alone hand me a gap this big or bigger? A p-value of 0.03 says: data this lopsided would show up only 3 percent of the time if B were no better than A. That is a surprising result under the assumption of no effect, surprising enough that most teams stop believing the “no effect” story and call the test significant.

So a low p-value is a measure of surprise. It is evidence against the idea that nothing is happening. It is not, and this matters enormously, a measure of how likely your variant is to be the winner.

The four misreadings that cost you

Here are the wrong interpretations, ranked by how much damage they do.

• “p equals 0.03, so there is a 97 percent chance B is better.” Wrong. The p-value assumes B is not better and measures your data’s surprise. The probability that B wins is a different quantity that only a Bayesian analysis produces.
• “p equals 0.03, so there is a 3 percent chance my result is a fluke.” Wrong, and subtly so. The 3 percent is the chance of this data under no effect, not the chance your specific conclusion is mistaken.
• “A tiny p-value means a big, important effect.” Wrong. p-values measure evidence, not magnitude. With a huge sample, a meaningless 0.1 percent lift can post p below 0.001. Read the effect size too.
• “The test is not significant, so the variants are the same.” Wrong. Failing to reach significance means you did not find enough evidence, often because your sample was too small. Absence of proof is not proof of absence.

The core rule - a low p-value means your data is surprising if nothing is going on, nothing more.

Why 0.05 is a choice, not a law

The 0.05 threshold is a convention that stuck, not a fact of the universe. Setting the bar at 0.05 means you accept roughly a 1-in-20 false-positive rate on tests where there is genuinely no effect.

That number has consequences the moment you run more than one test. Check 20 metrics on a change that does nothing, and on average one will cross 0.05 by pure luck. This is the multiple-comparisons problem, and it is why disciplined teams pre-register a single primary metric and treat the rest as secondary. It is also why peeking - repeatedly checking an in-progress test and stopping at the first significant reading - is so destructive. Every peek is another roll of the dice, and a test built for a 5 percent false-positive rate can blow past 20 or 30 percent once you peek and stop opportunistically. Our statistical significance guide walks through this in detail.

p-values, significance, and confidence intervals

These three are the same idea wearing different clothes.

• The p-value is the probability of your data under no effect.
• Statistical significance is just the p-value clearing a threshold, usually 0.05.
• The confidence interval is the flip side - a 95 percent interval that excludes zero corresponds to a two-sided p-value below 0.05.

The confidence interval is often the more useful of the two because it shows you the plausible range of the true effect, not just a pass-fail. A result can be significant yet have an interval spanning “barely worth it” to “genuinely great,” which tells you far more than “p equals 0.04” alone.

How the tools present it

Different engines surface the p-value differently, and the good ones protect you from the misreadings above.

Optimizely built its reputation on a stats engine that decides significance for you and is designed to stay valid under continuous monitoring, so you are less exposed to the peeking trap than with a raw p-value you check by hand. GrowthBook lets you run a frequentist engine with p-values and confidence intervals or switch to a Bayesian probability-to-beat readout, plus sequential testing for safe monitoring. Statsig ships sequential testing and CUPED so you can watch tests live and reach conclusions on less traffic. For picking between them, see the best experimentation platforms roundup.

The bottom line

A p-value is surprise-under-no-difference, not the probability your variant wins. Treat 0.05 as a tolerance you chose, not a truth you discovered. Never read a small p-value as a large effect - check the confidence interval and the effect size to see whether a real result is also worth shipping. Do not peek and stop early unless your tool supports sequential testing. Get those habits right and the p-value becomes what it should be - a guard against fooling yourself, not a rubber stamp for whatever you hoped was true.