Cash-back retail media
The Cash-Back Claim Test That Rewarded Pre-Existing Purchase Intent
A realistic measurement scenario showing how a useful advertising channel can still produce a misleading read when the comparison group is wrong.
Archetype: Local cash-back marketplace where users claim an offer before visiting a retailer
Bias mechanism: The test group had to claim an offer before buying, while the control group did not complete an equivalent intent-revealing step. Claiming the offer is partly a measure of pre-existing purchase intent.
Business model pressure
Retailers pay for traffic, sales, or measurable visits from cash-back users who claim offers before transacting.
Advertiser proof claim
The campaign report can show higher purchase rates among claimers. The hidden issue is that claiming is itself a selection event, not just an exposure event.
Statistical result
| Metric | Naive read | Stratified read | Modeled benchmark |
|---|---|---|---|
| Conversion lift | 26.0 pts | 2.0 pts | 1.0 pts |
| Incremental conversions | 13,029 | 1,000 | 502 |
| Incremental revenue | $938,104 | $71,977 | $36,132 |
| ROAS after media cost | 4.47x | 0.34x | 0.17x |
The naive analysis is 13.0x larger than the stratified estimate in this worked example.
The advertiser-facing story
Claimers convert at a much higher rate than non-claimers, so the campaign looks like a powerful traffic and sales driver.
What broke
The act of claiming filters for people already planning a purchase or already close to a store. If the control group does not have an equivalent intent gate, engagement becomes a disguised confounder.
Better design
Randomize offer availability before the claim step, compare claim-eligible users rather than claimers, and separately report claim-rate, visit-rate, and incremental purchase effects.
Propensity-strata audit
The adjusted estimate compares treated and control users within similar treatment-propensity strata. That does not replace a randomized holdout, but it shows how much of the naive result was carried by who entered treatment.
| Propensity stratum | Users | Treated conversion | Control conversion | Within-stratum lift |
|---|---|---|---|---|
| 1 | 2,750 | 6.1% | 6.4% | -0.3 pts |
| 2 | 16,827 | 9.4% | 9.1% | 0.3 pts |
| 3 | 10,744 | 14.3% | 13.5% | 0.8 pts |
| 4 | 3,375 | 19.4% | 18.5% | 0.9 pts |
| 5 | 951 | 22.9% | 20.9% | 2.0 pts |
| 6 | 2,253 | 33.0% | 31.3% | 1.7 pts |
| 7 | 8,593 | 42.2% | 40.1% | 2.1 pts |
| 8 | 19,724 | 49.5% | 47.5% | 2.0 pts |
| 9 | 22,540 | 60.8% | 56.7% | 4.1 pts |
| 10 | 2,243 | 71.0% | 68.2% | 2.8 pts |
Takeaway
A strong advertiser report should not stop at attributed conversions. It should show how the comparison group was built, whether treatment users had stronger prior intent, and how much of the result survives a better counterfactual.
Use this case in a readout review
After reading the failure mode, move from diagnosis to a better question set with these practical tools.
- Claim confidence rubricScore whether the readout supports strong, qualified, or tentative claim language.
- Measurement method selectorChoose the method that best fits the question: holdout, matched market, MMM, survey readout, or QA.
- Audience selection bias checklistAudit whether the measured group revealed stronger intent before the campaign could cause an outcome.