Sample Ratio Mismatch

What it means

You designed a 50/50 test but the data shows 52,000 users in control and 48,000 in treatment. Is that just chance? An SRM check runs a statistical test, usually a chi-squared test, against the intended ratio. If the observed split is far more lopsided than random variation would explain, you have a sample ratio mismatch. It is a red flag that the experiment plumbing is broken rather than a result to interpret.

Why it matters

An SRM almost always means the groups are no longer exchangeable, so the whole comparison is untrustworthy no matter how significant the headline number looks. Common causes are a redirect or bot filter that drops one arm more than the other, assignment that fires before a page fully loads, or logging that misses events for one variant. Because these mechanisms correlate with the metric you care about, an uncorrected SRM can manufacture a fake winner. Detecting it early saves you from shipping a change on rotten evidence.

In practice

A built-in SRM check is a sign the platform takes validity seriously. GrowthBook runs SRM checks as part of its stats engine, flagging tests where the arms did not fill as designed before you read the result. The fix is never to reweight the data; it is to find the bug, often in how the rollout percentage assigns traffic, and rerun the A/B test clean. See how to run an A/B test for the checks worth running before you trust a result.