Retargeting measurement
The Retargeting Campaign That Counted Cart Returners As New Demand
A realistic measurement scenario showing how a useful advertising channel can still produce a misleading read when the comparison group is wrong.
Archetype: Retargeting campaign measuring attributed return visits and purchase matchbacks
Bias mechanism: The treatment group is enriched for people who recently browsed, compared, saved, or abandoned a cart, so the campaign is credited for conversions that many users were already likely to complete.
Business model pressure
A retargeting program earns budget by showing efficient return visits, attributed purchases, and recovered carts from people who recently showed product interest. The advertiser wants to know whether the media created incremental demand or mainly reminded high-intent visitors who were already moving toward purchase.
Advertiser proof claim
A dashboard reports strong attributed ROAS from exposed visitors who returned and bought within the lookback window, but the core question is how many of those visitors would have returned through direct, email, organic search, price comparison, or saved-cart behavior without the retargeting impression.
Statistical result
| Metric | Naive read | Stratified read | Modeled benchmark |
|---|---|---|---|
| Conversion lift | 29.6 pts | 0.9 pts | 0.6 pts |
| Incremental conversions | 22,460 | 720 | 455 |
| Incremental revenue | $2,156,186 | $69,091 | $43,690 |
| ROAS after media cost | 5.26x | 0.17x | 0.11x |
The naive analysis is 31.2x larger than the stratified estimate in this worked example.
The advertiser-facing story
The report looks persuasive because exposed visitors are already close to purchase. They viewed product pages, compared options, saved items, abandoned carts, or came from high-intent sessions before the retargeting touch. When those visitors return and buy, the attributed result can look like recovered demand.
What broke
Retargeting eligibility is not neutral assignment. It is built from signals that predict conversion even without another ad. If the comparison group includes people who did not recently show the same product intent, the readout confuses audience selection and natural return behavior with media-caused lift.
Better design
Define eligible visitors before launch, randomize a protected holdout inside the same recent-intent pool, keep suppression and frequency rules fixed, and report lift against mature conversion windows. The readout should separate return visits, recovered carts, attributed purchases, and incremental purchases.
Propensity-strata audit
The adjusted estimate compares treated and control users within similar treatment-propensity strata. That does not replace a randomized holdout, but it shows how much of the naive result was carried by who entered treatment.
| Propensity stratum | Users | Treated conversion | Control conversion | Within-stratum lift |
|---|---|---|---|---|
| 2 | 12,388 | 9.1% | 7.9% | 1.2 pts |
| 3 | 23,108 | 12.3% | 12.3% | -0.0 pts |
| 4 | 10,772 | 16.9% | 17.6% | -0.6 pts |
| 5 | 2,226 | 26.9% | 23.3% | 3.6 pts |
| 6 | 213 | 28.4% | 35.6% | -7.1 pts |
| 7 | 3,417 | 42.0% | 43.9% | -1.9 pts |
| 8 | 22,919 | 54.0% | 53.0% | 1.0 pts |
| 9 | 46,150 | 66.5% | 64.7% | 1.9 pts |
| 10 | 4,807 | 77.7% | 77.3% | 0.4 pts |
Takeaway
A strong advertiser report should not stop at attributed conversions. It should show how the comparison group was built, whether treatment users had stronger prior intent, and how much of the result survives a better counterfactual.
Use this case in a readout review
After reading the failure mode, move from diagnosis to a better question set with these practical tools.
- Claim confidence rubricScore whether the readout supports strong, qualified, or tentative claim language.
- Measurement method selectorChoose the method that best fits the question: holdout, matched market, MMM, survey readout, or QA.
- Attribution window and conversion lag checklistReview whether lookback windows and lag patterns credited demand that was already likely to return.