Incrementality testing

Incrementality testing answers the only question that matters about ad spend: would these sales have happened anyway? Attribution tells you which ad a buyer saw before purchasing. It cannot tell you whether the ad caused the purchase or just got in front of someone who was always going to buy. Incrementality measures the difference between a world with the ads and a world without them, which is the actual definition of whether the spend worked.

This is the same logic as a controlled experiment, applied to media. You create a group exposed to the ads and a comparable group that isn’t, and you measure the gap. The gap is the incremental effect. Everything the platforms report is correlation; this is the causal version, and it routinely shows that a chunk of “attributed” revenue would have arrived with no ads at all.

The main methods

• Geo holdouts. Turn a channel off in a set of regions, leave it on in comparable ones, and compare sales. The cleanest method for a brand big enough to have regional spread, because geography gives you a real control group without needing the platform’s cooperation.
• Ghost ads and PSA tests. Within the platform, the control group is shown an unrelated or public-service ad where they’d have seen yours. Both groups were eligible and targeted, so the only difference is your creative, which makes it a true holdout rather than a comparison of dissimilar people.
• Conversion lift studies. The platforms’ own incrementality tools. Useful, but you’re trusting the platform to mark its own homework, so treat the result as a reason for scepticism rather than the last word.

Why DTC specifically needs this

Because DTC lives and dies on paid efficiency and the reported numbers are inflated. A brand can look ROAS-positive on every channel and still be lighting money on fire, if most of the “attributed” sales were going to happen regardless. Incrementality is how you separate the spend that’s actually buying growth from the spend that’s buying credit for organic demand. It’s the most rigorous, and most neglected, measurement work in the channel.

Where it breaks

• It needs scale and patience. Geo tests need enough regions and enough sales per region to clear the noise, which is a sample-size problem. Small brands often can’t run a clean one and have to accept messier reads.
• Contamination. If the holdout group is reachable through another channel - the region you turned Facebook off in still sees your TV ad - the test understates the true effect. Clean isolation is hard.
• One-off, not always-on. An incrementality test is a snapshot under the conditions you ran it. Creative fatigue, seasonality and auction shifts mean a channel’s incremental value drifts, so the test has a shelf life and needs rerunning.