Incrementality

Randomized lift test readout checklist

Published May 31, 2026. Status: evergreen source page.

A randomized lift test can be the cleanest evidence a campaign team receives. It can also be misread when the report hides assignment details, treatment leakage, weak outcomes, noisy intervals, or segment cuts chosen after the result was visible.

Use this checklist after a conversion-lift, brand-lift, store-lift, or account-level experiment arrives. The question is not whether the report contains a positive number. The question is whether the design still estimates the effect the decision needs.

What must be visible first

Readout field	What to check	Why it matters
Decision	The budget, audience, creative, market, or vendor decision the test was meant to inform.	A precise test can still be irrelevant if it answers the wrong business question.
Assigned unit	User, household, account, store, market, device, or another unit, plus why that unit could be separated.	The effect is only as credible as the separation between treatment and control.
Randomization rule	How eligible units were assigned, when assignment happened, and whether assignment stayed fixed.	Post-assignment filtering can turn a randomized test into a selected comparison.
Eligible population	Inclusion rules, exclusions, customer status, geography, device coverage, and any minimum activity rules.	The result should not be generalized beyond the population that could enter the test.
Exposure delivery	Impressions, reach, frequency, budget, pacing, creative, placement, and delivery quality by treatment group.	A weak or uneven treatment can produce a weak result for operational reasons, not because the idea had no effect.
Control protection	Suppression logs, control exposure, overlapping campaigns, partner delivery, and household or account spillover.	A contaminated control group shrinks or distorts the measured contrast.
Outcome source	Conversion, revenue, margin, lead quality, brand metric, store visit, or retention definition, with the data owner and lag.	Outcome data quality determines whether the lift number measures the decision outcome or a convenient proxy.
Uncertainty	Confidence or credible interval, sample base, minimum detectable effect, and the pre-stated decision threshold.	A positive point estimate is not enough when the interval is too wide for the decision.

Assignment integrity checks

Balance before exposure

Treatment and control should look similar on pre-period outcomes, customer mix, geography, device coverage, sales status, and prior engagement. Randomization does not excuse skipping balance checks when the sample is small or filtered later.

Stable membership

The report should show whether units stayed in their assigned cells. Removing low-activity users, unmatched records, or difficult geographies after assignment can make the treatment group look cleaner than the control.

Equal observation

Treatment and control outcomes should be captured with the same rules. If buyer-side matching, store reporting, CRM status, or survey recruitment differs by cell, the readout may be measuring observation quality.

Control leakage

Ask how many control units received the tested exposure through audience extension, retargeting, reseller media, shared households, sales outreach, or another campaign. The readout should state how leakage was estimated.

Lift math that should be shown

A credible readout gives enough arithmetic to reconstruct the claim. Relative lift alone is not enough because it hides the base rate, the absolute effect, and the commercial scale.

Metric	Useful readout	Common weak version
Treatment and control base	Assigned units, analyzed units, exposed units, matched units, and exclusions by cell.	Only reporting total conversions or total survey completes.
Base rates	Treatment outcome rate and control outcome rate with the same denominator definition.	Only reporting a percentage lift.
Absolute lift	Percentage-point difference between treatment and control.	Only reporting relative lift, which can sound large when the base rate is small.
Incremental outcome count	Incremental conversions, qualified leads, revenue, margin, or brand responses with the readout window.	Reporting attributed outcomes as if they were incremental outcomes.
Interval and power	Uncertainty interval, minimum detectable effect, and whether the test could detect the action threshold.	Calling a noisy positive estimate a win without showing precision.
Commercial threshold	Lift translated into margin, payback, qualified pipeline, or another pre-stated decision hurdle.	Stopping at statistical lift even when the effect is too small to matter.

Segments and timing

Segment reads should be planned before the result is visible or clearly labeled exploratory.
Segment bases should show treatment and control counts, not only the strongest lift cells.
Timing windows should include the expected response lag and avoid stopping on the first favorable week.
High-value outliers should be handled with a rule set before the readout, especially for revenue or margin outcomes.
If the test reports new customers, repeat customers, leads, or store visits separately, each outcome needs its own denominator and decision limit.

Decision language by result pattern

Pattern	Careful interpretation	Budget posture
Positive effect, clean assignment, narrow interval, clears margin hurdle.	The test supports scaling within the tested population, creative, bid, season, and outcome window.	Scale gradually and monitor decay, saturation, and audience expansion effects.
Positive point estimate, wide interval, does not clear the pre-stated threshold.	The test is not strong evidence even if the direction is favorable.	Repeat with more power or lower the decision stakes.
Near-zero estimate with a narrow interval around the action threshold.	The tested setup likely did not create a commercially meaningful effect.	Pause, redesign, or reallocate unless another strategic objective is documented.
Positive aggregate but weak or contradictory planned segments.	The average may be hiding heterogeneous effects or a fragile subgroup result.	Retest the segment with a locked plan before concentrating spend.
Control leakage, changed eligibility, or unequal outcome capture.	The contrast no longer cleanly estimates the intended counterfactual.	Use the report operationally, but downgrade causal language.

Questions for the readout call

What was the exact assignment unit, and how was assignment protected after launch?
How many treatment and control units were removed after assignment, and why?
How much control exposure or cross-cell leakage was observed or estimated?
Are the treatment and control base rates, absolute lift, relative lift, and interval all visible?
Which segment cuts were pre-stated before the result, and which were discovered afterward?
What decision threshold did the team agree to before seeing the lift number?

Takeaway

A randomized test deserves more trust than ordinary attribution, but it still needs inspection. If assignment, control protection, outcome capture, and uncertainty are visible, the result can guide a bounded decision. If those pieces are missing, the lift number may be a polished version of a weaker comparison.