Experimentation and A/B Testing

From rollout to experiment

In the last chapter you learned to ramp a change to a slice of traffic and watch how that slice behaves. An A/B test is that same machinery pointed at a different question. A progressive rollout asks “is this safe to expand?” An experiment asks “is this actually better than what we had?” Both split users into groups with a flag. The difference is that an experiment randomizes assignment and then compares a chosen metric between the groups rigorously enough to make a decision you can defend.

This matters because intuition is a terrible product manager. Teams ship redesigns that feel obviously better and watch conversion drop. They kill features that looked flat but were quietly retaining a valuable segment. Experimentation replaces “we think” with “we measured,” and feature flags are the cleanest way to run one because the flag already controls exactly who gets which variant. The what is A/B testing primer is worth a read if the concept is new.

The anatomy of a flag-based experiment

Every experiment has the same moving parts.

A hypothesis. Write it before you build. “Moving the signup button above the fold will increase signups” is testable. “Making the page nicer” is not. A good hypothesis names the change, the metric, and the direction you expect.

A randomized assignment. The flag hashes a stable user key into variants - control and one or more treatments - so assignment is random but sticky. The same user always sees the same variant for the life of the test. This is the exact sticky bucketing from the progressive delivery chapter, now used to form comparison groups.

An exposure event. You log the moment a user is actually assigned a variant, not merely when the page loads. This is subtle and important. If you count users who never reached the tested surface, you dilute your effect with people who could not possibly have been affected. Log exposure at the point of evaluation.

A metric. Pick one primary metric before you start. Guardrail metrics like error rate, latency, and revenue per user ride alongside so a treatment that lifts clicks but tanks revenue gets caught.

An analysis. After enough data, you compare the metric between groups and ask whether the difference is larger than what random noise would produce.

Getting the statistics right

This is where most homegrown experimentation goes wrong, so it deserves real attention.

Sample size is not optional. Before launching, calculate how many users you need to detect the smallest effect you would care about. Underpowered tests either miss real wins or, worse, produce noisy “wins” that evaporate in production. If you cannot reach the sample size in a reasonable window, the test is not worth running.

Peeking inflates false positives. In a classic fixed-horizon test, you commit to a sample size and only look at significance once, at the end. If you check every day and stop the first time p drops below 0.05, your real false positive rate is not 5 percent, it can be 20 or 30. Either commit to the horizon, or use a platform with sequential testing designed for continuous monitoring. The statistical significance in A/B testing post explains why this happens.

Significance is not importance. With enough traffic, a 0.1 percent lift becomes statistically significant and still is not worth the maintenance cost of the feature. Look at the effect size and its confidence interval, not just the p-value.

Run for a full cycle. Weekday and weekend users differ. Run at least one complete week so your sample is not skewed by whichever days the test happened to catch.

Here is the honest workflow for a checkout button test:

1. Hypothesis: a sticky checkout bar lifts completed purchases by at least 2 percent.
2. Power calculation says you need 40,000 users per arm. At current traffic that is nine days.
3. Launch a 50/50 flag split, log exposure at the checkout page, track completed purchases as primary and revenue per user as a guardrail.
4. Do not look at significance until day nine or the sequential engine flags it.
5. Read the effect size and interval, check the guardrail, then ship or roll back the flag.

How the real tools do it

The value of a dedicated platform is that assignment, exposure logging, and correct statistics are handled for you.

Statsig pairs feature flags and experiments in one system, so the same flag that ramps a feature also produces an experiment readout with sequential testing and automatic guardrail monitoring. If you are already ramping with flags, promoting a rollout to a measured experiment is nearly free.

GrowthBook is the leading open-source option. It layers experiment analysis on top of your existing data warehouse, so your numbers come from your own source of truth rather than a vendor pixel, and it supports both Bayesian and frequentist readouts.

PostHog bundles experiments with product analytics and session data, which is handy when you want to see not just whether a variant won but what users did around it. See its experiments feature for specifics.

Optimizely is the enterprise incumbent, strong on marketing-side and web experimentation with a mature stats engine. The best experimentation platforms roundup compares these trade-offs in depth, and how to do A/B testing walks through a full test end to end.

Common mistakes

• Testing everything. Experiments cost traffic and time. Reserve them for decisions where being wrong is expensive and intuition is genuinely split.
• No guardrails. A treatment can win your primary metric while quietly hurting revenue or performance. Always watch guardrails.
• Ignoring segments carefully. Slicing results by segment after the fact until you find a significant one is how you fool yourself. Pre-register the segments you care about.
• Never shipping the loser cleanup. When the test ends, one variant becomes permanent and the flag becomes debt. That handoff is exactly the subject of the next chapter.

Key takeaways

• An experiment is a feature flag plus randomized assignment plus honest metric comparison.
• Log exposure at the point of evaluation, not the page load, or you dilute your effect.
• Calculate sample size up front, avoid peeking, and read effect size rather than the bare p-value.
• Dedicated platforms exist because the statistics are easy to get subtly and expensively wrong.

Every experiment leaves a flag behind, and every rollout you finished in the last chapter did too. Those flags pile up. The final chapter, managing flag debt, is about cleaning them up before they turn your codebase into a minefield.