Guide - Diagnosing a Metric Movement
Date: 2026-09-28
One number moved. Pin down exactly what moved, check it isn’t noise, date it, split it into the parts that moved, then rule out the measurement layers cheapest first — timezone, definition, collection, identity, attribution — before anyone is allowed to explain it with customer behaviour. A cause counts only when it explains the size, the date and the shape of the move, and predicts something you can check.
Diagnosing a metric movement is the ordered investigation that takes a changed number to a confirmed cause — either a measurement artefact or a real change in behaviour — with evidence for each.
Where it sits:
- When a number looks wrong — the lookup: symptom → likely cause. Use it at step 5
- Guide - Auditing a Tracking Plan — the whole measurement surface, on a schedule
- This guide — one number, now, as a procedure
Same retailer as the other guides: 120,000 sessions and 2,880 orders a month — about 27,700 sessions and 665 orders a week, 2.40% conversion.
The running case
Monday morning, the weekly dashboard:
conversion rate, last week 2.19% previous week 2.40% −8.6%
Someone has already suggested it’s the new product page design that went live a fortnight ago. Every step below is applied to this.
The procedure
0 PIN IT DOWN which metric, which tool, which period, compared with what
1 IS IT NOISE? numerator and denominator, each against its normal variation
2 DATE IT the day it started, and the shape — step, slope or spike
3 DECOMPOSE numerator or denominator; which factor; mix or rate
4 WHAT CHANGED the change log for that date — deploys, tags, campaigns, stock
5 WALK THE LAYERS timezone → definition → collection → identity → attribution
6 BEHAVIOUR only now: is something genuinely different about customers?
7 CONFIRM size, date, shape, and one prediction checked
8 WRITE IT UP verdict, evidence, the unexplained remainder, the fix
The order is cheapest-first and most-likely-first at the same time. Measurement changes are more common than behaviour changes, and cheaper to check. Investigations that stall usually started at step 6.
0. Pin it down
Write the movement as one sentence before investigating anything. Vague statements send people after different numbers:
MOVEMENT STATEMENT
metric session conversion rate = orders ÷ sessions, analytics tool,
all traffic, bots filtered
value 2.19%, week Mon 6 – Sun 12
compared with 2.40%, previous week (same weekdays, same filters)
also checked same week last year: 2.37% — so not a seasonal dip
source of alert weekly dashboard, Monday
- Which tool — analytics, ad platform and order system each have their own conversion rate, and they disagree for known reasons — Tool Discrepancies
- Compared with what — adjacent periods are the least comparable: a week after a sale, a month with an extra weekend. Check the same period last year as well — Seasonality
- Is the definition written down? If not, the first finding may be that two people mean different things by the metric — Metric Design
1. Is it noise?
Test the numerator and the denominator separately. A rate can move because either moved, and they have different normal variation.
The quick check for a count, treating it as roughly Poisson — where the standard deviation of a count is about its square root:
normal weekly orders ≈ 665
standard deviation √665 ≈ 25.8 = 3.9% of 665
two standard deviations ≈ 7.8% ← a move smaller than this is
unremarkable in a single week
last week: orders 662 −0.4% well inside the band
sessions 30,179 +9.0% session counts normally vary
by a few per cent week to week
← outside it
So the conversion move is real as a movement in the metric — but it’s the denominator that moved, not the orders. That already argues against the product page theory: a worse page would lose orders, and orders didn’t fall.
In plain terms: 665 orders a week will naturally wobble by 25 or so either way, so a week of 662 is nothing. What’s unusual is 2,500 extra sessions producing no extra orders.
Counterintuitive, and why: “beyond normal variation” means the number genuinely changed, not that customers did. Step 1 separates signal from noise; it says nothing about whether the signal is behaviour or measurement. That’s steps 3–6.
Real week-to-week variation is usually wider than the square-root rule suggests — promotions, email sends and weather add their own swings — so where you have a year of history, use its actual spread instead. The method: Anomaly Detection. The same idea as an interval: Confidence Intervals.
2. Date it
Daily, not weekly. The day it started and the shape it took narrow the causes more than anything else:
sessions orders conversion
Mon 3,986 95 2.38%
Tue 3,986 96 2.41%
Wed 4,442 93 2.09% ← step
Thu 4,442 95 2.14%
Fri … Sun 4,442/day ~94 ~2.12%
| Shape | Usually means |
|---|---|
| Step — changes on one day, stays | A deploy, a tag publish, a config change, a consent-banner change. Something switched |
| Slope — drifts over weeks | Browser privacy changes eroding identity, gradual mix change, slow creep in bot traffic |
| Spike — changes and comes back | A campaign, an outage, a bot burst, a one-day tracking failure |
| Seasonal-shaped | Nothing; compare with last year |
A step on Wednesday. Steps are almost always something someone did, which makes step 4 short — Metric Drift.
3. Decompose
Split the move into the parts that make it up, largest first — Metric Decomposition has the method:
previous last week change
sessions 27,692 30,179 +9.0% ← carries the move
orders 665 662 −0.4%
conversion 2.40% 2.19% −8.6% = 0.996 ÷ 1.090 = 0.914
Then mix versus rate, across every dimension you’d normally segment by. On device, both segments fell by the same proportion — not mix. On traffic source, one line jumps out:
source previous last week
paid search 6,120 6,150
organic 8,980 9,020
…
referral: own domain 0 2,279 ← new on Wednesday, 0 orders credited
to it... but paid search orders
are down 18%
Sessions arriving from the site’s own domain, starting Wednesday, roughly 2,300 in five days — and paid search, the channel most customers start checkout from, losing orders at the same moment.
4. What changed on that date
Check every log for the day the step started — Annotation and Change Logs:
- Code deploys and tag manager publishes
- Consent banner or cookie changes
- Marketing — campaigns launched, emails sent, budgets changed
- Pricing, stock, delivery changes
- Platform and vendor status pages; browser releases
- Analytics configuration — filters, session settings, channel definitions
The deploy log for Wednesday: checkout moved to checkout. subdomain. That’s the candidate.
If nothing is logged, that’s a finding too — the organisation needs a change log, and until it has one, every investigation starts with asking around.
5. Walk the layers
Even with a candidate, walk the layers in order — confirming a cause is cheaper than finding out later there were two. Each has a fastest check; the symptom table maps the specific signs:
| Layer | Question | Fastest check | Here |
|---|---|---|---|
| Timezone and date | Same boundaries in both periods and both tools? | compare the same period in the order system — Timezones and Date Boundaries | order system: 668 orders, flat. Boundaries fine |
| Definition | Is it the same metric as last week? | analytics config history; filters; channel rules — Metric Drift | unchanged |
| Collection | Is each event firing, once, at the right moment? | event counts by day; live debugging — Instrumentation Debugging | purchase events fire; counts match the order system as usual |
| Identity and sessions | Are sessions and users counted the way you think? | sessions per user; users vs sessions trend — Sessionisation · User Counting | sessions +9%, users +2% — sessions are splitting |
| Attribution | Is credit landing where it used to? | channel shares of orders — Attribution Models | orders moved from paid search to “own domain” |
Found at identity. Moving checkout to a subdomain without telling the analytics tool the two domains are one site means every visit to checkout starts a new session — referred by the site itself — and the return to the confirmation page starts another. Each checkout visitor now counts as two extra sessions, and their order is credited to the self-referral instead of the channel that brought them. [CHECK: the setting name for cross-domain measurement and referral exclusion in the analytics tool in use]
6. Behaviour — only now
If steps 1–5 all come back clean, the change is probably real, and the question becomes what’s different about customers or the offer. The sweep:
- Traffic mix — a campaign or channel change bringing different visitors. Mix/rate decomposition on source and landing page says so directly — Metric Decomposition · Channel Mix
- The site — a release that changed the experience; check the metric by the templates it touched
- The offer — price, delivery, promotions, returns policy; stock — Stockouts and Availability
- The outside — competitor sales, weather, news, paydays, a bank holiday — Seasonality
- Regression — a record week followed by an ordinary one isn’t a fall — Regression to the Mean
In the running case this step isn’t reached, and the product page theory never needed testing.
7. Confirm
A cause is confirmed when it explains the size, the date and the shape — and predicts something you haven’t looked at yet.
prediction if the subdomain is the cause observed
size extra sessions ≈ checkout visits × 2 ≈ 1,140 checkout visits
Wed–Sun × 2 = 2,280 ✓
date starts on the deploy day Wednesday ✓
shape a step, not a slope step ✓
prediction 1 order system flat 668 orders ✓
prediction 2 paid search orders fall; "own domain" rises −18% / new source ✓
prediction 3 conversion with self-referral sessions removed 2.37% ≈ normal ✓
returns to normal
The third prediction does the most work: it’s a number you couldn’t have produced by fitting a story to the data you’d already seen.
Anything left unexplained goes in the write-up. If the correction had brought conversion back to 2.30% rather than 2.37%, something else was also happening, and a clean story would hide it.
8. Write it up
Short, and the same shape every time, so the next investigation can find it:
INCIDENT conversion −8.6%, week of 6 Oct verdict: ARTEFACT
What moved analytics session conversion 2.40% → 2.19%
Cause checkout moved to subdomain (deploy, Wed 8 Oct) without
cross-domain measurement: each checkout visit split into
two extra self-referred sessions
Evidence sessions +9.0% vs users +2%; step on deploy day; order
system flat (668); self-referral source 0 → 2,279;
corrected conversion 2.37%
Unexplained none material
Impact conversion understated and paid search under-credited
from 8 Oct until fixed; attribution reports for the period
unreliable
Fix cross-domain configuration; referral exclusion for own domain
Annotate dashboard annotation 8 Oct → fix date, "sessions inflated"
Prevention tracking checks added to the release checklist for any
domain or checkout change
Not the cause the product page redesign — orders never moved
The last line matters. The redesign would otherwise carry the blame in everyone’s memory, and the next person to propose reverting it will cite this week.
Annotate the dashboard for the affected dates, or next year’s comparison will rediscover the dip as a mystery — Annotation and Change Logs. And add the check to whatever release checklist exists: the cheapest investigation is the one that never has to happen — Data Quality Monitoring · Alerting on Metrics.
How investigations go wrong
- Starting with behaviour. The redesign theory arrived before anyone looked at the orders
- Checking only the rate. Numerator and denominator move for different reasons; looking at the ratio alone hides which
- Weekly data for dating. The step was on a Wednesday; the weekly number blurred it across two weeks
- Adjacent-period comparison only, so a seasonal dip reads as a fall
- Stopping at the first plausible cause without checking size, date and shape — and missing a second cause hiding in the remainder
- Trusting one tool. The order system settled most of this in one query
- Not writing down what was ruled out, so the same theory resurfaces next month
The short version
- Write one sentence: metric, tool, period, comparison
- Check numerator and denominator against normal variation separately
- Date it by day; step, slope or spike
- Decompose: which factor, then mix versus rate on every segmenting dimension
- Read every change log for the start date
- Walk timezone → definition → collection → identity → attribution
- Only then behaviour
- Confirm on size, date, shape and one fresh prediction; write down what’s unexplained and what was ruled out
Related: When a number looks wrong for the symptom-to-cause lookup · Guide - Auditing a Tracking Plan for the whole surface · Guide - Ecommerce CRO §3 for the measurement gate before any CRO work. Domain map: Analytics MOC.