Guardrail Metrics in Experimentation - What They Are and How to Set Them (2026)

Every experienced experimenter has a story about the test that won and still hurt the business. A new checkout flow lifts conversion three percent, ships, and two weeks later someone notices refunds crept up and the payment API is timing out more often. The primary metric was honest. It just was not the whole story. Guardrail metrics exist so that the rest of the story cannot be ignored.

This post explains what a guardrail is, why it is tested differently from your primary metric, and how to set thresholds that actually protect you rather than fire on noise. For the one-line version, the guardrail metric glossary entry has you covered. This is the long form.

What a guardrail metric actually is

A guardrail metric is a measure you watch alongside an experiment to make sure a change does not damage something you refuse to sacrifice. It is not a success measure. You do not run the test hoping the guardrail improves. You run it hoping the guardrail stays flat, and you build a tripwire in case it does not.

The distinction from your primary metric matters. Your primary metric, sometimes called the Overall Evaluation Criterion, is the one number you commit to deciding the test on, and you want it to go up. A guardrail is the opposite posture - a metric you expect to hold steady and are only checking for harm. That difference in intent changes how you test it, which is the part most teams get wrong.

Why guardrails need a different statistical test

Here is the subtle point. For a primary metric you run a normal two-sided significance test, asking whether the variant produced a real lift rather than noise. If the result is not significant, you did not learn enough to ship.

A guardrail flips that logic. You are not trying to prove the guardrail improved. You are trying to be reasonably confident it did not get meaningfully worse. That is a non-inferiority test, and it needs a threshold - a degradation you are willing to tolerate. You might decide that up to 20 milliseconds of extra latency is acceptable, but anything worse blocks the ship. The test then asks whether you can rule out a degradation larger than that margin.

This is why a guardrail that shows a small, non-significant dip is usually fine, while an ambiguous guardrail - one where you cannot rule out real harm because the test was underpowered - should make you nervous. A flat-looking guardrail with wide confidence intervals is not reassurance. It is missing information. For how those intervals are read, see confidence intervals in A/B testing.

The three families of guardrail

Most useful guardrails fall into three groups.

• Technical health. Page load time, API latency, error rate, crash rate, timeout rate. These catch a variant that wins on the surface but degrades the experience underneath. A heavier hero image that lifts engagement while adding 400 milliseconds of load time is a trade you never agreed to.
• Business health. Refund rate, unsubscribe rate, support ticket volume, chargebacks. These catch wins that borrow from the future - a pushier upsell that lifts conversion today and refunds next month.
• Trust and count integrity. This is the one people forget. A sample ratio mismatch check is effectively a guardrail on the experiment itself. If your 50/50 split arrived as 52/48, the assignment is broken and every other number is suspect, guardrail or not.

Keep the list short - typically three to six - and choose them during design, not after you see the result. Adding a guardrail after a surprising result is how you rationalise a decision you already made.

Organisation-wide vs experiment-specific guardrails

The strongest experimentation cultures run two layers. A small set of universal guardrails applies to every experiment automatically - usually latency, error rate, and a top-level business metric like revenue per user or retention. On top of that, each experiment adds one or two guardrails specific to the surface it touches.

The universal layer is what stops a single team optimising a local number at the expense of the whole company. It is also why guardrails live best inside the platform rather than a spreadsheet. When the metric is attached to the experiment, the tool can flag a regression the moment it appears rather than waiting for a human to think to look.

Where the tooling helps

Guardrails are only useful if the platform computes them for you and warns automatically.

Statsig lets you attach guardrail metrics to any experiment and, being an experimentation-first platform built by ex-Facebook engineers, treats them as a first-class part of the readout rather than an afterthought. Its flag and config checks are unlimited and free on every tier, with analytics events as the meter.

Optimizely carries guardrail monitoring inside its long experimentation heritage, with a free Rollouts tier and sales-led pricing above it. Its always-valid Stats Engine means you can watch guardrails continuously without the peeking penalty that would otherwise inflate false alarms.

PostHog exposes secondary and guardrail metrics next to the primary in its experiments product, which sits inside a broader analytics suite so the metric you are guarding is often already instrumented.

Common mistakes

• Testing a guardrail like a primary metric. Demanding significance to trust it inverts the logic - you end up ignoring a real regression just because it did not clear a two-sided bar.
• No threshold. A guardrail without a tolerable-degradation margin is just a number you stare at. Decide the margin in advance.
• Too many guardrails. Watch twenty and, at a 5 percent false-positive rate each, one will trip by chance almost every test. Short and deliberate beats exhaustive.
• Choosing them after the result. Guardrails belong in the design phase, alongside the primary metric and the minimum detectable effect.

The bottom line

Guardrail metrics turn win rate into net value. A test that wins on the primary metric and holds every guardrail is a real win. A test that wins on the primary metric while tripping a guardrail is a trade, and you should have agreed to it before launch or not at all. Set them during design, test them for non-inferiority against a threshold you chose in advance, and keep the list short.

To put guardrails in their place, see how to design an A/B test for the full pre-launch workflow, how to choose metrics for A/B testing for picking the primary metric they sit beside, and how to run an A/B test for executing the plan.