Metric Drift
Date: 2026-08-16
The definition changed and the chart kept drawing. Nothing breaks, no alert fires, and the trend line now compares two different measurements as though they were one — which is worse than a gap, because a gap is visible.
What it is
Metric drift is a change in how a metric is computed, without a change in its name or a break in its reporting, so historical and current values are silently non-comparable.
conversion rate
4% ┤ ╭──────────
3% ┤────────╯
2% ┤
└──┬─────┬─────┬────
Jan Apr Jul
↑
bot filter added
nothing marks this
The step looks like performance. It’s a definition change, and by the time anyone asks, the person who made it has forgotten.
What causes it
Ranked by how often it’s the answer:
- A tool migration. Different session rules, different bot filtering, different attribution defaults. Every metric steps at once — Tool Discrepancies
- A consent banner change, which alters the measured population — Consent Management
- Bot filtering added, removed or changed — Bot and Internal Traffic
- A filter someone added for one report that became the default
- The denominator changing — sessions to users, or a session timeout adjustment — Sessionisation
- A tracking fix. Correcting a double-firing event reduces the count, correctly, and the chart shows a decline — Double Counting
- Attribution model or window changed, which rewrites every channel simultaneously — Attribution Models
- A currency, tax or refund handling change in revenue — Revenue Metrics
The pattern: most drift comes from fixing something. That’s what makes it insidious — the improvement and the apparent decline arrive together.
Why it’s worse than an outage
An outage is visible. Data stops, someone notices, it gets fixed and the gap is obvious in the chart forever.
Drift produces a complete, plausible, wrong series. Decisions get made on it. Worse, it usually gets attributed to something real — a campaign, a redesign, a seasonal effect — so it generates a false belief that outlives the data.
Detecting it
- Step changes with no business cause. A metric that moves on a Tuesday and stays there is a definition change until proven otherwise
- Two metrics moving together that shouldn’t. If sessions, users and conversion rate all step on the same date, the common cause is measurement
- Ratios behaving oddly. Orders flat while conversion rate rises means the denominator moved
- Reconciliation drift. The gap between analytics and the order system widening is the earliest signal, which is why it’s worth tracking as a standing metric — Guide - Auditing a Tracking Plan
Preventing it
- Annotate everything. Deploys, container publishes, consent changes, tool migrations, filter changes — on the same timeline as the metrics. This is the single highest-value practice here and it costs nothing — Annotation and Change Logs
- Write the definition down, with an owner, so a change is a change to a document rather than to someone’s query — Metric Design
- Version deliberately. If a definition genuinely must change, either restate history under the new definition or start a new metric with a new name. Never silently apply a new definition going forward — that’s the thing that creates the invisible seam
- Announce it in advance. A fix that will reduce a reported number should be flagged before it lands, in writing, or the fix gets blamed for the decline
When you find it
Three options, and the choice is about credibility rather than accuracy:
| Option | When |
|---|---|
| Restate history under the new definition | You can recompute — warehouse-side data, and the change matters |
| Annotate and leave both | You can’t recompute. Mark the seam clearly on every chart |
| Start a new metric | The definitions are different enough that one series would mislead |
What you must not do is nothing. An unmarked seam means every future comparison across that date is wrong, indefinitely, and the person who makes that comparison won’t know.