Evidence standard
Causal claim review protocol
Organization byline: Measurement Press editorial desk. This page does not list an individual author or reviewer. Read the editorial method and corrections policy.
About 9 min read
Causal language is where weak evidence often becomes expensive. A story says an event caused a change, a dashboard says a campaign drove sales, or a report says a tactic improved an outcome. The reader needs a fast way to decide whether the verb has earned that confidence.
This protocol is for any claim that uses words like caused, drove, led to, changed, increased, reduced, prevented, influenced, or worked. It helps separate observed sequence from causal evidence before the claim becomes a headline, meeting takeaway, renewal argument, or budget recommendation.
The six-question pass
1. What exactly changed?Name the outcome, population, time window, geography, and unit of analysis. A causal claim is not reviewable until the claimed change is specific enough to inspect.
2. What is the proposed cause?Separate one event, policy, campaign, message, product change, or source action from the background conditions around it. If the cause is vague, the evidence standard is already lower.
3. What would have happened anyway?Name the counterfactual: prior trend, peer group, holdout, matched market, untreated users, comparable records, model baseline, or survey control. If no counterfactual is visible, the claim should not sound settled.
4. Was the comparison chosen before results were visible?Post-hoc comparisons can still teach, but they are weaker than planned comparisons. Watch for convenient chart windows, selective markets, cherry-picked segments, and omitted failed outcomes.
5. What else changed at the same time?List concurrent events, seasonality, pricing, distribution, creative, product changes, reporting changes, data-quality shifts, eligibility changes, and audience mix changes. Each plausible alternate explanation should narrow the claim.
6. What language can the evidence support?Choose the verb after the design is visible. "Followed," "correlates with," "is consistent with," "suggests," "estimated lift," and "supports a causal read" are different claims.
Claim-type matrix
| Claim type | Strongest first check | Common overclaim | Safer readout language |
|---|---|---|---|
| News timeline | Prior state, timestamped records, mechanism, lag window, and concurrent events. | Sequence is treated as cause. | "The event preceded the change; the record should show whether the change was already underway." |
| Before-and-after chart | Full source series, denominator, chart window, seasonality, peer comparison, and revision risk. | A visible break is treated as proof of impact. | "The series changed after the event, but the comparison is not yet strong enough for causal language." |
| Public-record count | Record universe, inclusion rules, missing cases, rate base, and comparable period. | A count change is treated as population change. | "The reported count changed in this record universe; the rate base determines how broad the claim can be." |
| Campaign attribution | Exposure rule, attribution window, prior intent, comparison group, and credited outcome definition. | Credited conversions are treated as incremental conversions. | "The report credits observed outcomes to the campaign; lift still needs a protected comparison." |
| Lift test | Assignment, holdout protection, leakage, primary outcome, minimum detectable effect, and uncertainty. | A noisy or contaminated test is treated as decisive. | "The test estimates lift under this design and uncertainty range." |
| Model output | Controls, priors, calibration evidence, sensitivity, response curves, and omitted drivers. | Predictive fit is treated as causal proof. | "The model supports a planning range if the assumptions and calibration evidence hold." |
| Survey or brand study | Population, sample source, exposed/control balance, wording, field dates, and respondent quality. | Respondent movement is treated as market or sales movement. | "The survey reports a difference among respondents under this field design." |
Primary source trail and a simulated claim gate
Source-to-decision links: The CDC interpretation guidance lists chance, selection bias, information bias, confounding, and investigator error as alternative explanations to check before calling an association causal. AAPOR’s disclosure standards provide a transparency checklist for survey and content-analysis claims. Both are adapted as review aids here and do not validate the simulated test or any real campaign result.
The protocol’s proof burden is not just a writing preference. The Google Ads Conversion Lift documentation separates a treatment group from a control group and defines the difference as incremental lift; ordinary attributed conversions do not answer the same counterfactual question. Google’s Conversion Lift setup guidance also treats study power as dependent on the conversion action, budget, duration, estimated lift, and holdback. For broader statistical review, the Federal Judicial Center’s Reference Guide on Statistics asks whether a study is designed to investigate causation, how units were collected and measured, and whether the result can be generalized. For survey-based claims, AAPOR’s Transparency Initiative disclosure standards require the sponsor, researcher, population, dates, wording, sampling frame, and coverage limits to be available.
Boundary: These references define questions a reviewer should ask; they do not validate a source packet, guarantee causal identification, or turn the simulated numbers below into a real campaign result. CDC interpretation guidance is public-health guidance adapted here as a general bias check: chance, selection bias, information bias, confounding, and investigator error can all create an apparent association.
Simulated worked example - not a real campaign, study, or forecast: assume a pre-registered test assigns 2,000 eligible people to treatment and 2,000 to control, uses the same 30-day conversion definition, and protects the control group from the ad. Treatment records 120 conversions; control records 90.
| Check | Calculation | What the evidence can support |
|---|---|---|
| Treatment rate | 120 / 2,000 = 6.0% | Observed conversion rate under the simulated treatment condition. |
| Control rate | 90 / 2,000 = 4.5% | Observed rate for the simulated counterfactual condition. |
| Absolute difference | 6.0% - 4.5% = 1.5 percentage points | The unadjusted between-group difference under the stated design. |
| Relative difference | 1.5% / 4.5% = 33.3% | A relative comparison whose denominator must be named. |
| Illustrative incremental count | 2,000 x 1.5% = 30 | A planning quantity for the treatment-sized group, not proof beyond the test population. |
The narrow conclusion is: “Under the simulated assignment, conversion was 1.5 percentage points higher in treatment, equivalent to 30 additional conversions at the treatment-group size.” It is not yet safe to write “the campaign drove 30 conversions” if assignment was not protected, the groups differed before treatment, the outcome window was changed after results appeared, exposure leaked into control, or another event affected only one arm. The conclusion also changes if the outcome is a proxy, the denominator excludes eligible records, the sample is too small for useful uncertainty, or the result is generalized beyond the tested population. A real review should attach the assignment record, pre-period balance check, outcome definition, maturity date, confidence or credible interval, exclusions, and any deviations from the registered plan.
Simulated packet acceptance gate
This second pass is a review decision, not a new result. Assume the fictional packet contains the rates and assignment description above, but does not include a pre-period balance check, a concurrent-event log, a mature lag analysis, or an uncertainty interval.
| Gate | Result | Why it matters |
|---|---|---|
| Outcome and denominator are named | Pass | The 30-day conversion definition and 2,000-person group sizes are explicit. |
| Comparison is defined | Pass | The packet identifies treatment, control, and the protected control condition. |
| Pre-period balance and concurrent events | Hold | Without these checks, group differences or another event can mimic lift. |
| Outcome maturity and lag | Hold | A fixed window is not enough if conversions arrive after the readout closes. |
| Uncertainty and study power | Hold | The point difference needs an interval or power context before it carries a high-confidence claim. |
| Claim scope and disclosure | Pass | The conclusion is limited to the simulated population and states that it is not a production result. |
Reproducible decision: 3 of 6 gates pass = 50.0%. The packet can support a bounded statement about the simulated between-group difference, but it should be held at descriptive or directional language until the three held gates are supplied. A reviewer can reproduce the score by counting the six rows; adding one missing gate changes the score by 1 / 6 = 16.7 percentage points, but does not by itself prove causality.
Downgrade triggers
Move a causal claim down at least one confidence level when one of these conditions is present. Move it down more than one level when several appear together.
- The evidence shows only what happened after launch, publication, announcement, or exposure.
- The comparison group was chosen after the favorable result was visible.
- The affected group was already different before the claimed cause occurred.
- The denominator, starting level, or eligible population is hidden.
- The outcome changed at the same time as pricing, seasonality, product availability, measurement rules, or audience mix.
- The reported result is a proxy for the decision outcome: attention instead of sales, recall instead of persuasion, clicks instead of lift, or quoted reaction instead of evidence.
- The uncertainty interval, subgroup base, sample source, or exclusion rule is missing.
Evidence request menu
| If the claim depends on | Ask for | Decision rule |
|---|---|---|
| Chronology | Timestamped records, prior-state evidence, lag expectation, and concurrent-event list. | Use causal wording only if the mechanism and timing survive alternate explanations. |
| A trend chart | Underlying data, full period, denominator, seasonality check, peer trend, and revision status. | Do not let a chosen window carry the whole claim. |
| A campaign readout | Holdout, matched comparison, mature outcome window, exposure logs, leakage check, and pre-period intent. | Keep attribution language separate from lift language. |
| A model | Model specification, controls, priors, calibration tests, sensitivity checks, and uncertainty ranges. | Use model output as a bounded estimate, not a standalone proof. |
| A survey | Population, sample source, field dates, exact wording, weighting, subgroup bases, and control balance. | Keep the conclusion inside the respondent universe and measured construct. |
| A public record | Record owner, inclusion rules, missing universe, rate base, comparable period, and revision rules. | State what the record shows before claiming what actually changed in the wider world. |
Decision note template
Use this short note before forwarding, publishing, renewing, or escalating a causal claim.
Claim:Write the exact sentence and mark the causal verb.
Observed fact:State what the source clearly shows without interpretation.
Counterfactual:Name the comparison used, or state that no credible comparison is visible.
Alternate explanations:List the strongest timing, selection, denominator, mix, measurement, or incentive explanations.
Confidence level:Choose decision-grade, strong but bounded, directional, weak signal, or unsupported frame.
Allowed wording:Write the strongest sentence the evidence can support today.
Use it with the rest of the library
For media claims, pair this protocol with the timeline and event-order framing checklist, before-and-after trend checklist, public records and denominator checklist, media claim audit worksheet, and evidence-to-claim language matrix.
For advertising measurement, pair it with the measurement method selector, incrementality test plan template, randomized lift test readout checklist, MMM causal validity checklist, and campaign readout QA checklist.
Takeaway
Causal wording should be earned, not inherited from a chart, quote, dashboard, or vendor summary. When the counterfactual is weak, keep the sentence descriptive. When the design is strong, state the limits that make the causal claim useful.