What Is a Holdout Group? Measuring the Cumulative Impact of Experiments (2026)

Individual A/B tests have a blind spot. Each one tells you whether a single change beat the old version, but they never tell you whether all your shipped changes, taken together, actually moved the business. Ten tests that each claimed a 2 percent lift do not reliably add up to a 20 percent gain - effects overlap, decay, and sometimes were noise to begin with. A holdout group is how mature teams close that gap, by keeping a clean slice of users away from everything and watching what really changed.

What a holdout group is

A holdout group is a set of users deliberately excluded from one or more experiments or feature rollouts and kept on the existing experience. While everyone else receives your stream of shipped changes, the holdout stays on the baseline. Later you compare the two and read the cumulative effect of everything the holdout missed.

The key word is cumulative. A holdout is not asking about one change. It is asking, over this whole period, did the sum of what we shipped actually improve the metric against a group that got none of it? That is a question no single experiment can answer.

Holdout group vs control group

This is the distinction people trip over, so it is worth being precise. The two look similar - both are users kept on the old experience - but they operate at different scopes.

	Control group	Holdout group
Scope	One experiment	Many experiments or all of them
Lifetime	The test’s duration	Weeks to a quarter or more
Question	Did this test win?	Did everything combined move the metric?
Baseline	No change, this test only	No recent changes at all

A control group is the baseline inside a single A/B test. A holdout sits above individual tests and stays out of many at once. A control group validates one decision; a holdout validates a whole program of decisions.

The types of holdout

Two flavours cover most uses.

• Global holdout. A small, fixed percentage of your user base - often 1 to 5 percent - excluded from every experiment in a given area for a defined window, typically a quarter. At the end you compare their metrics against the exposed population to see the combined impact of that quarter’s shipping. This is the classic “did all our experimentation deliver” measurement.
• Long-term or per-feature holdout. A slice kept off a specific shipped feature for an extended period, to measure effects that only appear over time - retention, habit formation, or the fade of a novelty effect that made the initial test look better than the durable reality.

Both exist because short experiments over-credit changes whose benefit shrinks once the novelty wears off, and holdouts are the tool that catches that decay.

Why the cumulative number matters

Here is the uncomfortable truth holdouts expose. Across a quarter of experiments, some “wins” were false positives, some real effects overlapped so their gains do not stack, and some decayed. If you naively sum the claimed lifts you get a fantasy number. The holdout measures reality - the actual difference between users who got your changes and users who did not - which is almost always smaller than the sum of the parts, and occasionally reveals that a celebrated program moved nothing at all.

That honesty is the entire point. It is also why the top experimentation cultures treat the global holdout as a governance metric, reported to leadership, rather than a nice-to-have.

The cost, stated plainly

A holdout is not free. The users in it are deliberately kept on an older, presumably worse experience for the whole window, so you forgo the benefit of your improvements for that slice. The group also has to be small enough not to cost much yet large enough to stay statistically powered - which for a small effect can be a real tension. And it takes engineering discipline to ensure those users are consistently excluded from the right experiments over time, without leaking in through inconsistent assignment. A broken holdout produces a sample ratio mismatch style problem, where the comparison is quietly invalid.

For most teams shipping many changes, the trade is worth it, because there is no other clean way to know whether the program as a whole is working.

Where the tooling helps

Holdouts are an advanced feature, so they appear in experimentation-first and analytics-integrated platforms.

PostHog supports holdout groups in its experiments product, letting you reserve a percentage of users from a set of tests inside the same analytics suite where the metrics live. Statsig carries holdout functionality within its full experimentation engine, built by ex-Facebook engineers to mirror a rigorous internal stack, with unlimited free flag and config checks on every tier. GrowthBook, being warehouse-native, can model long-term holdout comparisons against your own event data, though it asks for someone comfortable with the statistics.

The bottom line

A holdout group measures what individual A/B tests cannot - the real, combined, durable impact of everything you shipped, against a baseline that got none of it. It is the difference between believing the sum of your test wins and knowing your true top-line movement. Keep it small, keep it consistent, and treat the cumulative number as the honest scorecard for your whole experimentation program.

To connect this to the rest of the workflow, see how to design an A/B test for individual test rigour, guardrail metrics in experimentation for protecting against regressions, and what is an experimentation platform for the tooling that manages holdouts at scale.