How to Choose Metrics for A/B Testing - Primary, Secondary and the OEC (2026)

You can run a flawless experiment - clean randomisation, correct sample size, no peeking - and still reach a worthless conclusion, because the metric you chose to measure was the wrong one. Metric selection is where most A/B tests are actually won or lost, and it happens before a single user is bucketed. This guide walks the decision step by step.

Step 1 - Start from the outcome you actually care about

Before naming any metric, write down the real goal in a sentence. More paying customers. Higher retention. Faster activation. This anchors everything, because a metric is only good insofar as it tracks that goal.

The trap here is measuring what is easy instead of what matters. Clicks are easy to count. Revenue is harder. A change that lifts an easy proxy while doing nothing for the real outcome is a false win, and choosing that proxy as your metric is how you manufacture one. Name the outcome first so every metric you pick can be judged against it.

Step 2 - Choose exactly one primary metric

This is the most important rule in experiment design. Pick the single metric that decides the test, and commit to it before launch. In the literature this primary metric is called the Overall Evaluation Criterion, or OEC.

The reason to pick one is blunt statistics. Track ten metrics, let any of them declare victory, and with a 5 percent false-positive rate on each you have roughly a 40 percent chance that at least one moves “significantly” by pure luck. Choose which metric to report after seeing the data and you will always find a winner. Fixing one primary metric up front is what stops you mining noise for a story. For the mechanics of why that false-positive rate compounds, see p-value in A/B testing explained.

A good primary metric is:

• Sensitive. It moves enough within a realistic sample to be detectable. A metric that only shifts over months is useless in a two-week test.
• Aligned with value. Moving it should genuinely matter. This rules out vanity metrics.
• Hard to game. It should not be trivially inflatable by a change that helps nothing else.

Step 3 - Avoid vanity metrics

A vanity metric is a number that looks good and moves easily but does not connect to the outcome from Step 1. Page views, raw click counts, and time on page are the usual suspects. A more aggressive interstitial can lift clicks while raising bounce and hurting conversion. Time on page can rise because users are confused, not engaged.

If a metric can go up while your business gets worse, it is not fit to be a primary metric. The test for vanity is simple - ask whether you would genuinely be happy shipping a change that moved only this number and nothing else. If the answer is no, demote it to secondary.

Step 4 - Structure secondary metrics to explain, not decide

Secondary metrics are the supporting cast. They exist to help you understand why the primary moved, not to serve as backup sources of victory. If your primary metric is checkout conversion, useful secondaries include add-to-cart rate, cart abandonment, and average order value, because together they tell you the mechanism.

The discipline is that a secondary metric moving in your favour never rescues a flat primary. The moment “well, the primary did not move but look at this other number” enters the conversation, you have started fishing. Secondaries inform the narrative; they do not overturn the verdict.

Step 5 - Add guardrails for what must not break

Finally, name the metrics you do not expect to improve but refuse to let regress - latency, error rate, crash rate, refund rate, unsubscribes. These are your guardrail metrics, and they catch the winning variant that quietly ships a regression. Choose three to six, keep a couple universal across every experiment, and set a tolerable-degradation threshold for each.

The full structure looks like this.

Role	Question it answers	Decides the test?
Primary metric (OEC)	Did the change win?	Yes, and only this
Secondary metrics	Why did the primary move?	No, explains only
Guardrail metrics	Did anything important break?	Can block a win

Step 6 - Beware metric dilution and multiple comparisons

Two quieter traps deserve a mention. Metric dilution happens when your metric includes users the change could never affect - measuring sitewide conversion when your test only touched the pricing page drowns a real effect in unrelated traffic. Scope the metric to the population the change can actually move.

Multiple comparisons is the flip side of Step 2. Every extra metric or segment you allow to declare significance raises the odds of a false positive. If you truly need to test several, correct for it or, better, pre-declare the one that counts. This is also why the minimum detectable effect you set should reflect the primary metric specifically, since it drives your sample size.

Where the tooling fits

The platform should make the primary-secondary-guardrail structure explicit rather than dumping a wall of numbers.

Statsig organises experiments around a primary metric with secondary and guardrail metrics attached, reflecting the rigorous internal experimentation stack its ex-Facebook founders modelled it on. Flag and config checks are unlimited and free on every tier.

GrowthBook is warehouse-native, so your metrics are defined once against your own BigQuery, Snowflake or Databricks data and reused across every experiment - which keeps metric definitions consistent, the thing spreadsheets always get wrong.

PostHog ties experiment metrics directly to its product-analytics events, so the primary metric is usually already instrumented before you design the test.

The bottom line

Metric selection is design work, not analysis work. Name the real outcome, commit to one primary metric before launch, keep secondaries for explanation and guardrails for protection, and never let an easy number stand in for a valuable one. Get that right and the analysis becomes almost mechanical.

Next, see how to design an A/B test for where metric choice sits in the full workflow, guardrail metrics in experimentation for the protection layer, and statistical significance in A/B testing for reading the primary metric once the test is done.