Sample Ratio Mismatch (SRM) - The Signal That Invalidates Your A/B Test (2026)

Sample ratio mismatch is the closest thing A/B testing has to a smoke alarm. When it goes off, you stop reading results and start looking for a fire, because a mismatched traffic split almost always means the two groups you are comparing are no longer comparable. SRM is the single most important test-validity check most teams have never heard of, and ignoring it is how a broken experiment gets shipped as a winner.

Sample ratio mismatch (SRM) is when the observed split of users across your variants differs from the split you configured, by more than random chance can explain. You asked for 50/50, the data came back 50.8/49.2, and that gap is too large to be luck. That is SRM. See the sample ratio mismatch glossary entry for the one-line version.

Why a mismatched split breaks everything

Randomised assignment is the whole point of an A/B test. It is what lets you claim that any difference in outcomes was caused by the variant and not by some pre-existing difference between the groups. When the split is off, that guarantee evaporates.

If users are being lost from one bucket, the ones that remain are a filtered, non-random sample. Maybe the slower variant drops impatient users before they get counted, leaving it with a more patient, higher-converting crowd. Now the variant looks like it “won,” but you measured survivorship, not the change you made. A test with SRM has not measured your variant - it has measured whatever is corrupting the split. No amount of statistical significance rescues it, because significance assumes the randomisation held.

How to detect SRM - the chi-square check

The detection method is a chi-square goodness-of-fit test. You compare the number of users you actually observed in each variant against the number you expected from your configured ratio.

The statistic is straightforward:

chi-square = sum over variants of (observed - expected)^2 / expected

Work a concrete example. You ran a 50/50 test and collected 100,000 users. You expected 50,000 per variant, but you observed 50,800 in A and 49,200 in B.

(50,800 - 50,000)^2 / 50,000  =  640,000 / 50,000  =  12.8
(49,200 - 50,000)^2 / 50,000  =  640,000 / 50,000  =  12.8
chi-square = 25.6

With one degree of freedom, a chi-square of 25.6 corresponds to a p-value of roughly 0.0000004 - less than one in a million. A gap that size does not happen by chance on a true 50/50 split. That test has SRM, and the correct action is to discard the result, not to report the “winner.”

Because you are checking this on every experiment, you do not want to cry wolf on healthy tests. That is why the alarm threshold is deliberately strict. Many practitioners flag SRM only when the p-value drops below 0.0005 or 0.001, far below the usual 0.05 significance bar, so that normal random variation never trips it.

What actually causes SRM

Finding SRM is easy. Finding the cause is the real work. The usual suspects:

• Inconsistent bucketing. Assignment code that does not hash a stable identifier deterministically can put the same user in different buckets or skip users entirely.
• Redirect and latency loss. A variant that redirects or loads more slowly drops users before the tracking event fires, so it silently loses traffic.
• Tracking fires at different points. If variant A logs exposure on page load and variant B logs it after an animation, B loses everyone who bounced during the animation.
• Bots and crawlers. Automated traffic that hits one arm disproportionately skews the counts.
• Post-assignment filtering. Removing users after they were bucketed - excluding a segment, dropping “invalid” sessions - can strip users unevenly if the filter correlates with the variant.

The common thread is that users are gained or lost in a way that is correlated with the variant itself. Random loss that hits both arms equally does not cause SRM. Loss that favours one arm does.

Where the tooling helps

You should not be running chi-square tests by hand on every experiment. This is exactly the kind of guardrail a serious platform automates.

GrowthBook runs SRM checks automatically as part of its stats engine, alongside CUPED, sequential testing and Bayesian analysis. Because it is warehouse-native and queries your own BigQuery, Snowflake or Databricks, it checks the split against your source-of-truth event data. It is the most statistically complete of the open-source options, with the trade that you need an instrumented warehouse and someone comfortable with the stats.

Statsig was built by ex-Facebook engineers to mirror a rigorous internal experimentation stack, and health checks including sample-ratio diagnostics are part of that heritage. Flag and config checks are unlimited and free on every tier, with analytics events as the meter. It is now owned by OpenAI as of September 2025.

Optimizely built its name on making experiment results trustworthy, with a stats engine designed to hold up under continuous monitoring. It has a free Rollouts tier, though its paid experimentation pricing is entirely sales-led.

SRM in the wider validity picture

SRM is one of a small set of checks that decide whether a result deserves to be believed at all, and it sits before significance in the pipeline. A test can look gorgeously significant and still be worthless if the split was broken. Check the sample ratio first, then read the p-value - never the other way around.

For the surrounding discipline, our statistical significance guide covers what a significant result does and does not claim, how to run an A/B test folds the SRM check into a full workflow, and if you are still choosing a platform, the best experimentation platforms roundup compares the stats engines head to head. The short version: a check that takes one query can save you from shipping a change that did nothing, and telling yourself it was a win.