Annotation and Change Logs
Date: 2026-08-16
A dated record of what changed, plotted against the metrics. It costs almost nothing and it’s the difference between “conversion dropped in March and nobody knows why” and a two-second lookup.
What it is
An annotation is a timestamped note attached to a metric timeline, recording something that happened which could explain a movement in it.
conversion rate
4% ┤ ╭────────
3% ┤────────╯
└──┬─────┬─────┬──
Jan Apr Jul
│
└─ 14 Apr: bot filter enabled
18 Apr: checkout redesign, 50% rollout
22 Apr: consent banner updated
Without those three lines the step is a mystery, and it will be attributed to whichever explanation is most available — usually the one someone is already invested in.
Why it’s the highest-value cheap practice
Because the cost of the alternative is unbounded. An unexplained step change generates a meeting, then an investigation, then a plausible wrong answer that enters institutional belief and gets acted on for years.
And the window for finding the real cause is short. Three weeks later the person who published the container change has forgotten, the deploy log has rolled over, and the campaign that started that Tuesday isn’t in anyone’s memory.
In plain terms: you get one chance to record why something moved, and it’s at the moment it moves. Everything after that is archaeology.
What to record
Anything that could move a number, which is more than people assume:
| Category | Examples |
|---|---|
| Deploys | Releases, feature flags enabled or ramped, rollbacks |
| Tracking | Container publishes, new or removed tags, SDK upgrades, filter changes |
| Definitions | Metric definition edits, attribution model or window changes, bot filtering |
| Consent | Banner redesigns, category changes, anything affecting acceptance |
| Commercial | Campaign starts and ends, price changes, promotions, stock events |
| External | Outages, platform incidents, algorithm updates, competitor activity, weather where it matters |
| Migrations | Replatforms, analytics property changes, domain changes |
The categories that get recorded least and cause most confusion are tracking and definitions — precisely because they’re the ones that produce a step with no business cause.
Where it lives
Two options, and they’re complementary rather than alternatives:
In the analytics tool. Most have an annotation feature. Immediate, visible where people look, and locked inside that tool.
As a table in the warehouse, joined to reporting. Portable, queryable, survives a tool migration, and can be filtered by category. This is the durable version — Warehouse-First Analytics.
If you only do one, do the warehouse table and surface it in dashboards. Annotations trapped in a tool you later leave are lost exactly when a migration makes them most valuable.
Making it happen
Discipline fails; automation doesn’t:
- Deploy hook. CI writes an annotation on every production release. Free after an afternoon’s work, and it covers the largest category
- Container publish webhook, where the tag manager supports it
- Flag changes logged automatically by the flag platform — Feature Flags
- Campaign starts pulled from the ad platforms’ APIs
- Manual for the rest, with one person responsible and a low bar for adding
The bar should be very low. A wrong annotation costs nothing; a missing one costs an investigation. Err towards recording.
Failure modes
- Only annotating incidents. Then you have a log of things that went wrong and no context for anything that went right
- Annotating in one tool only, so the warehouse and BI have no context
- No categories, so you can’t filter to “just tracking changes” when diagnosing a definition problem
- Free text with no structure. “Fixed the thing” is not an annotation
- No owner, so it happens for a month and then stops
Where it’s used
It’s the last step of nearly every diagnostic in the vault, which is why so much points here: Metric Drift (was it a definition change?), Tool Discrepancies (when did the ratio move?), Bot and Internal Traffic (when was the filter enabled?), Guide - Auditing a Tracking Plan (what preceded the volume step?), and the ordering rule in the symptom list.