Measurement science
Next measurement method after weak budget evidence
A budget worksheet is useful when it shows that the evidence is not strong enough for the action being requested. The next step is not to ask for a louder dashboard. It is to choose the measurement method that answers the missing question.
Picture a renewal meeting where the campaign report shows clean delivery, more site visits, and a few persuasive charts, but no proof of what would have happened without the spend. The buyer wants a budget increase, the publisher wants a renewal, and the analyst has to keep the decision inside the evidence. That is the moment this guide is for.
Use this guide after the budget decision evidence ladder or budget decision meeting worksheet flags a weak evidence lane. The choice usually comes down to four designs: MMM calibration, geo lift, randomized holdouts, or a brand study.
First name the gap
The method should match the missing counterfactual. If the report cannot show what would have happened anyway, ask which comparison would change the budget decision.
| Weak evidence lane | Best next method | Use when the decision is | Do not use it to prove |
|---|---|---|---|
| Planning assumptions across channels, seasons, or spend levels are weak. | MMM calibration. | How much budget should move across a portfolio, with ranges and assumptions visible. | That a single campaign caused a specific conversion path. |
| The action is market, region, store, or territory based. | Geo lift or matched-market test. | Whether a market-level change produced incremental outcomes beyond comparable markets. | That every market, season, or spend level will behave the same way. |
| The action needs person-level or household-level conversion lift. | Randomized holdout. | Whether exposed eligible users changed behavior against a protected control group. | That the result automatically generalizes beyond the assigned population and window. |
| The question is awareness, recall, consideration, or favorability. | Brand study. | Whether surveyed perception moved inside a defined population. | That perception movement caused sales, profit, or pipeline lift. |
Decision chooser
Translate the jargon immediately. MMM, short for marketing mix modeling, estimates how channels tend to move outcomes across time. Geo lift compares treated and untreated markets. A randomized holdout protects a control group before exposure. A brand study asks people whether awareness, recall, or consideration changed. None of these is automatically better. Each answers a different version of "compared with what?"
Use for portfolio planning
Choose this when the budget question spans channels or quarters and a model needs credible anchors. The evidence request should ask which experiments, geo tests, holdouts, or external priors calibrate the model, how sensitive the ranking is to those anchors, and where uncertainty changes the decision.
Use for market-level actions
Choose this when treatment can vary by market, store, region, or territory. The evidence request should ask for pre-period balance, matched controls, local shocks, spillover checks, outcome maturity, and the rule that decides whether the test changes the budget.
Use for controlled exposure questions
Choose this when eligible users, households, accounts, or devices can be assigned before exposure and the control can be protected. The evidence request should ask for assignment logs, suppression rules, leakage checks, base rates, minimum detectable effect, and uncertainty.
Use for perception outcomes
Choose this when the decision is about message memory, awareness, consideration, or favorability. The evidence request should ask for sample source, exposed-control balance, question wording, field dates, weighting, respondent quality, and the exact budget decision the survey can inform.
If the budget decision cannot wait
Sometimes the decision window arrives before the better method can be run. In that case, the action should stay below the evidence rung and the next evidence request should be explicit.
| Current evidence | Safer interim action | Next evidence request |
|---|---|---|
| Clean delivery and exposure, weak outcome evidence. | Repair setup, preserve clean inventory, or renew only with narrow operational language. | Outcome-quality review, baseline comparison, or randomized holdout if lift language is needed. |
| Observed response is strong, but no comparison is visible. | Renew cautiously or keep the same scope while asking for a stronger counterfactual. | Protected holdout, geo test, or baseline comparison that shows what would have happened anyway. |
| Directional comparison favors the campaign, but uncertainty is material. | Make a bounded budget-direction call only if the low end still clears the business threshold. | Uncertainty review, minimum detectable effect plan, and a preplanned test for the next material change. |
| Model or survey evidence supports the story, but method details are thin. | Use the result as directional context, not final budget proof. | MMM calibration detail or brand-study method disclosure before stronger language is used. |
Worked example: a renewal ask with a weak comparison
A contextual package finishes a quarter with clean delivery, strong attention signals, and more qualified visits than the prior period. The renewal request asks for a 30 percent budget increase. The problem is simple: the report shows response, but it does not show what comparable buyers, regions, or audiences would have done without the package.
| Finding in the report | What it actually supports | Next-method implication | Budget language today |
|---|---|---|---|
| Delivery, pacing, and destination quality were clean. | The campaign was operationally usable. | Do not run a stronger lift design until the clean setup is preserved. | Renew or repair the package on operational grounds. |
| Qualified visits rose during the flight. | Observed response moved in the right direction. | Ask for a counterfactual: matched market, randomized holdout, or calibrated baseline. | Call it directional response, not incremental lift. |
| The decision is whether to raise spend in selected regions. | The action is market-level. | Plan a geo lift test with pre-period balance and local-shock checks. | Hold the increase to a bounded test budget. |
| The sales team wants a broad "worked" claim. | The evidence is below the claim rung. | Require outcome maturity, uncertainty, and a threshold for spend movement. | Say the package earned a stronger next test. |
Method-specific evidence requests
| Method | Ask for | Reject as insufficient |
|---|---|---|
| MMM calibration | Calibration anchors, priors, controls, response curves, uncertainty ranges, holdout or geo evidence, and sensitivity to exclusions. | A high fit score or channel ranking without calibration and uncertainty. |
| Geo lift | Market eligibility, matching variables, pre-period levels and trends, local events, spillover risk, outcome window, and readout threshold. | A post-period market gap with no pre-period balance or local-shock check. |
| Randomized holdout | Assignment unit, eligible universe, suppression proof, treatment compliance, control leakage checks, base rate, power, outcome maturity, and interval. | A platform lift label without assignment, leakage, base-rate, and uncertainty detail. |
| Brand study | Sample source, exposed and control balance, survey timing, question wording, weighting, respondent quality, outcome meaning, and uncertainty. | A single lift percentage that does not show who answered or what changed. |
How to write the next action
Good decision language names the method and keeps the action inside its boundary. A useful sentence sounds like this:
Decision languageThe current readout supports a cautious renewal and a stronger next test. Because the budget question is market-level, the next evidence request is a matched-market geo lift plan with pre-period balance, local-shock checks, outcome maturity rules, and a threshold for whether the result changes spend.
If the method is MMM calibration, replace the market language with calibration anchors, priors, sensitivity, and planning ranges. If the method is a randomized holdout, name assignment, suppression, leakage, base rate, and uncertainty. If the method is a brand study, keep the claim to surveyed perception and do not use it as sales lift.
Pair with
Start with the measurement method selector when the method is still unclear. Use the budget decision evidence ladder to choose the evidence rung, the budget decision meeting worksheet to record the action, the MMM calibration evidence checklist for model anchors, the geo lift test design checklist for market tests, the randomized lift test readout checklist for holdout readouts, and the brand lift study readout checklist for survey evidence.