Metric Design
Date: 2026-08-16
A metric is a definition, not a number. Most disagreements about figures are disagreements about definitions nobody wrote down — and the definition is what makes the number reproducible a year later.
What it is
Metric design is specifying what a number counts precisely enough that two people computing it independently get the same answer.
The specification, and all seven fields are load-bearing:
name checkout conversion rate
numerator users with a purchase event
denominator users with a checkout_started event
window same session
filters excludes bots, internal IPs, test orders
timezone Europe/London
owner alex
Miss the denominator and you have two metrics. Miss the timezone and your daily numbers disagree with finance’s — Timezones and Date Boundaries.
What makes one worth having
Three tests, and a metric should pass all three:
Does it change a decision? If every plausible value leads to the same action, it’s decoration. The test isn’t whether it’s interesting — it’s whether you’d do something different at 3% versus 5%.
Can it be gamed, and what happens if it is? Every metric becomes a target. Time on page rises when the site is confusing. Add-to-cart rate rises if you make the button ambiguous. Design assuming someone will optimise it, and pair it with a guardrail that catches the perverse route — Guardrail Metrics.
Is it sensitive enough to move detectably? A metric that can’t shift measurably at your traffic can’t inform anything. Revenue per visitor needs 2.4× the traffic of conversion rate for the same relative change — Metric Sensitivity.
Choosing the denominator
The decision that causes the most trouble, because it’s usually made by default rather than deliberately.
| Denominator | Moves when | Use for |
|---|---|---|
| Sessions | The inactivity timeout or event volume changes | Traffic shape only — Sessionisation |
| Users | Identity, consent or browser policy changes | Most reporting, with caveats — User Counting |
| Exposed users | Only the tested population changes | Experiments — Randomisation Unit |
| Funnel entrants | Nothing but the funnel itself | Step conversion — Funnel Analysis |
Session-scoped rates are the trap. The denominator is a modelling artefact, so a variant that increases interaction reduces session count and raises the rate with no additional orders. Worked in Sessionisation.
Task-scoped denominators are the most robust available: checkout_started → purchase depends on no identity model and no timeout.
Rates over counts
Counts move with traffic, so a count going up tells you nothing about performance. Rates isolate the thing you influence.
The exception is anything you’re accountable for in absolute terms — revenue, orders, margin. Report both: the rate for diagnosis, the count for the business.
Ratio metrics need care beyond this — the variance of a ratio isn’t the variance of its parts, which affects significance testing — Ratio Metrics.
Where definitions rot
- Nobody wrote it down, so it’s reconstructed differently each time it’s queried
- The definition changed silently and the trend line kept rendering across the seam — Metric Drift
- Two tools, one name. “Conversion rate” in your BI tool and in GA4 are different metrics — Tool Discrepancies
- Filters accumulated informally — someone excluded a market for one report and it became the standard
- No owner, so nobody’s job it is to notice any of the above
The one thing to do
Write the seven fields down for every metric that reaches a decision-maker, in one place, and link to it from the dashboard.
That’s it. Not a semantic layer, not a governance programme — a page anyone can check. Most metric disputes end within a minute once both parties are looking at the same definition, and the ones that don’t end are real disagreements worth having.