Search advertising
The Search Campaign That Confused Intent Capture With Incrementality
A realistic measurement scenario showing how a useful advertising channel can still produce a misleading read when the comparison group is wrong.
Archetype: Search advertising platform selling sponsored results beside high-intent queries
Bias mechanism: The treatment group is enriched for shoppers who typed commercial or branded queries, so last-click conversion credit captures demand that already existed before the ad impression.
Business model pressure
Advertisers pay for clicks or conversions associated with sponsored search placements. The platform has a strong reason to show that paid clicks produce profitable conversions.
Advertiser proof claim
A dashboard reports high ROAS from paid search clicks, but the counterfactual question is how many conversions would have happened through organic results, direct navigation, or another unpaid path.
Statistical result
| Metric | Naive read | Stratified read | Modeled benchmark |
|---|---|---|---|
| Conversion lift | 28.5 pts | 0.6 pts | 0.6 pts |
| Incremental conversions | 22,862 | 492 | 481 |
| Incremental revenue | $2,194,732 | $47,227 | $46,214 |
| ROAS after media cost | 4.57x | 0.10x | 0.10x |
The naive analysis is 46.5x larger than the stratified estimate in this worked example.
The advertiser-facing story
The dashboard makes paid search look exceptionally efficient because conversions are measured after the shopper has already revealed commercial intent. The ad receives credit for a path that may have completed through organic search or direct navigation.
What broke
Treatment assignment is not random. Query intent, brand familiarity, prior site visits, and product urgency all affect both the chance of clicking the ad and the chance of buying. Comparing ad clickers with everyone else makes intent look like lift.
Better design
Use randomized ad suppression, geo experiments, brand-term holdouts, or query-level experiments. Report incremental conversions, not only attributed conversions, and separate brand, category, and conquesting queries.
Propensity-strata audit
The adjusted estimate compares treated and control users within similar treatment-propensity strata. That does not replace a randomized holdout, but it shows how much of the naive result was carried by who entered treatment.
| Propensity stratum | Users | Treated conversion | Control conversion | Within-stratum lift |
|---|---|---|---|---|
| 2 | 5,768 | 9.2% | 7.3% | 1.9 pts |
| 3 | 18,414 | 11.0% | 10.3% | 0.6 pts |
| 4 | 15,513 | 14.6% | 14.6% | 0.0 pts |
| 5 | 6,749 | 20.6% | 18.3% | 2.4 pts |
| 6 | 1,518 | 23.5% | 26.1% | -2.6 pts |
| 7 | 1,534 | 39.1% | 33.5% | 5.6 pts |
| 8 | 13,441 | 48.5% | 47.6% | 0.8 pts |
| 9 | 47,053 | 61.1% | 60.4% | 0.7 pts |
| 10 | 15,010 | 72.3% | 73.0% | -0.7 pts |
Takeaway
A strong advertiser report should not stop at attributed conversions. It should show how the comparison group was built, whether treatment users had stronger prior intent, and how much of the result survives a better counterfactual.
Use this case in a readout review
After reading the failure mode, move from diagnosis to a better question set with these practical tools.
- Claim confidence rubricScore whether the readout supports strong, qualified, or tentative claim language.
- Measurement method selectorChoose the method that best fits the question: holdout, matched market, MMM, survey readout, or QA.
- Incrementality test plan templateDesign a cleaner holdout or suppression test before treating captured intent as incremental demand.