Confidence Intervals in A/B Testing - How to Read Them Right (2026)

Most teams read an A/B test result as a single verdict - significant or not, up 4 percent or flat. The confidence interval quietly carries far more information than that, and once you learn to read it you stop asking the wrong question (“did it win?”) and start asking the right one (“how much did it win, and how sure are we?”). This guide makes the interval readable.

A confidence interval is a range of plausible values for the true effect, reported with a confidence level like 95 percent. Where a p-value gives you a pass-fail, the interval gives you the shape of the answer.

What the interval is telling you

Suppose your test reports a 4 percent lift with a 95 percent confidence interval of 1 percent to 7 percent. Two things are packed into that.

• The range is where the true lift plausibly lives. Your best single estimate is 4 percent, but the honest answer is “somewhere between 1 and 7 percent, probably.”
• The width is your precision. A tight interval like 3.5 to 4.5 percent means you have measured the effect sharply. A wide one like -2 to 10 percent means you barely know anything yet and likely need more data.

A confidence interval that does not cross zero is statistically significant - and it also shows you how big the effect might be, which a p-value never does. An interval of 0.2 to 0.4 percent is significant but tiny. An interval of 5 to 30 percent is significant and possibly huge. Same “significant” label, wildly different decisions.

The interpretation trap everyone falls into

Here is the misreading to unlearn. A 95 percent confidence interval does not mean there is a 95 percent probability the true value sits inside your specific interval.

In frequentist statistics the true lift is a fixed (if unknown) number. Your interval either contains it or it does not - there is no probability about this one interval. The 95 percent describes the procedure - if you reran the experiment many times and built an interval each time, about 95 percent of those intervals would capture the true value. It is a statement about the method’s long-run reliability, not about your one result.

If you genuinely want “95 percent probability the true lift is in this range,” that is a credible interval, which comes from Bayesian analysis. Credible intervals and confidence intervals often look similar on the same data, but they mean different things, and only the Bayesian one licenses the intuitive “probability it is inside” reading.

The overlap trap

A tempting shortcut is to look at the confidence interval for variant A’s conversion rate and variant B’s conversion rate, see that they overlap, and conclude the difference is not significant. This is wrong often enough to be dangerous.

Two intervals can overlap while the difference between the variants is still statistically significant. The correct object to inspect is the confidence interval of the difference - B minus A - not the two separate rate intervals. If that difference interval excludes zero, the result is significant, whatever the individual intervals appear to do. Good experimentation tools report the difference and its interval directly for exactly this reason, so you are not eyeballing two error bars and guessing.

Relative vs absolute lift

Intervals get reported in two units, and mixing them up quietly distorts every result.

	Baseline 5 percent, variant 5.5 percent
Absolute lift	+0.5 percentage points
Relative lift	+10 percent

Absolute lift is the raw percentage-point change. Relative lift expresses it as a fraction of the baseline. A 0.5-point absolute gain on a 5 percent base is a 10 percent relative gain - the same result, two very different-sounding numbers. When someone says “we lifted conversion 10 percent,” always ask whether they mean 10 relative percent (5 percent to 5.5 percent) or 10 absolute points (5 percent to 15 percent), because those are worlds apart. This same distinction drives your minimum detectable effect and therefore your sample size.

How the tools present intervals

GrowthBook reports the difference and its interval directly, and in Bayesian mode gives you a credible interval plus the probability a variant beats control - so you can read the intuitive version honestly. Statsig pairs its intervals with sequential testing and CUPED variance reduction, and CUPED specifically tightens intervals by removing pre-experiment noise, meaning you reach a usable width on less traffic. Optimizely built its stats engine to keep intervals valid under continuous monitoring, so the range you read mid-test is not lying to you the way a naive interval would. For a wider look, see the best experimentation platforms roundup and Statsig vs GrowthBook.

The bottom line

Read the confidence interval, not just the significance verdict. Its width tells you how precisely you have measured the effect, and whether it crosses zero tells you if the result is real. Do not read “95 percent” as the probability the truth is in your one interval - that is a credible interval, from Bayesian stats. Compare the interval of the difference, never two separate rate intervals. And always know whether a lift is relative or absolute before you celebrate. The interval is the richest number your test produces - once you read it fluently, the p-value becomes a footnote.