Geo lift testing
The Geo Lift Test That Mistook Seasonal Markets For Media Lift
A realistic measurement scenario showing how a useful advertising channel can still produce a misleading read when the comparison group is wrong.
Archetype: Matched-market geo lift campaign with uneven pre-period demand
Bias mechanism: The treated markets already had stronger category momentum, higher baseline demand, and more favorable timing before the campaign, so the readout treats market selection and seasonality as incremental media impact.
Business model pressure
A regional media campaign earns budget by showing that activated markets produced more sales than comparison markets. The buyer needs to know whether the campaign changed demand, not only whether the chosen markets were already on a better path.
Advertiser proof claim
A readout reports higher sales, stronger ROAS, and a clean-looking lift percentage in exposed markets, but the counterfactual question is whether similar markets with the same pre-period trajectory would have grown without the extra media.
Statistical result
| Metric | Naive read | Stratified read | Modeled benchmark |
|---|---|---|---|
| Conversion lift | 20.0 pts | 1.2 pts | 0.9 pts |
| Incremental conversions | 16,339 | 949 | 736 |
| Incremental revenue | $1,094,721 | $63,558 | $49,283 |
| ROAS after media cost | 1.30x | 0.08x | 0.06x |
The naive analysis is 17.2x larger than the stratified estimate in this worked example.
The advertiser-facing story
The campaign appears to work because treated markets outpace comparison markets during the readout window. A clean chart shows a post-launch gap, and the report converts that gap into incremental sales and ROAS.
What broke
The treated markets were not exchangeable with the comparison markets. They had stronger pre-period growth, better category conditions, cleaner distribution, or more favorable seasonal timing before media pressure increased. A post-period gap can therefore measure market choice as much as media effect.
Better design
Choose comparison markets before launch, balance pre-period levels and trends, document excluded markets, keep promotion and distribution changes visible, and run sensitivity checks against alternate matched controls. A useful readout should show whether the conclusion survives a better baseline.
Propensity-strata audit
The adjusted estimate compares treated and comparison observations within similar pre-period propensity strata. That does not replace a randomized geo holdout, but it shows how much of the naive result was carried by which markets entered treatment.
| Propensity stratum | Market-audience cells | Treated conversion | Control conversion | Within-stratum lift |
|---|---|---|---|---|
| 2 | 5,158 | 8.3% | 6.5% | 1.8 pts |
| 3 | 14,864 | 10.2% | 9.8% | 0.4 pts |
| 4 | 15,209 | 14.3% | 12.8% | 1.5 pts |
| 5 | 10,222 | 17.6% | 17.1% | 0.5 pts |
| 6 | 7,147 | 24.2% | 23.5% | 0.6 pts |
| 7 | 11,516 | 36.7% | 35.1% | 1.6 pts |
| 8 | 25,600 | 45.4% | 44.5% | 1.0 pts |
| 9 | 35,921 | 56.4% | 55.5% | 1.0 pts |
| 10 | 6,363 | 68.0% | 63.4% | 4.6 pts |
Takeaway
A strong geo-lift readout should not stop at a post-launch market gap. It should show how markets were selected, whether pre-period trends were balanced, which operational changes overlapped the campaign, and how much of the result survives alternate comparison sets.
Use this case in a readout review
After reading the failure mode, move from diagnosis to a better question set with these practical tools.
- Claim confidence rubricScore whether the readout supports strong, qualified, or tentative claim language.
- Measurement method selectorChoose the method that best fits the question: holdout, matched market, MMM, survey readout, or QA.
- Geo lift test design checklistBalance pre-period trend, seasonality, market exclusions, and alternate comparison sets.