Evaluation worksheet
Source and vendor evaluation worksheet
Organization byline: Measurement Press editorial desk. This page does not list an individual author or reviewer. Read the editorial method and corrections policy.
About 9 min read
Use this page in editorial meetings, analytics reviews, vendor calls, and campaign readouts when a source or report is asking for more confidence than the evidence may support.
The worksheet turns a broad claim into a few inspectable pieces: the source trail, denominator, comparison class, causal design, uncertainty, and the fact that would most change the decision.
Use it when: a dashboard, research deck, public article, sponsored report, or campaign readout is moving toward a decision before the source trail is clear. The point is to leave with a bounded claim, a named weak field, and a next evidence request.
What the worksheet should make visible
A source review is strongest when it separates the persuasive sentence from the evidence beneath it. Before scoring, write down the exact claim, the decision it is meant to support, and the closest inspectable source. Then check whether the denominator, comparison class, source incentives, and uncertainty are visible enough for the decision being made.
| Pressure in the room | Field to expose first | Why it matters |
|---|---|---|
| A public claim sounds settled because several sources repeat it. | The original source trail and whether later citations add independent confirmation. | Repeated summaries can make one thin origin look like consensus. |
| A percentage, rate, or lift number is driving confidence. | The denominator, starting level, eligible universe, and excluded rows. | A strong-looking change can disappear when the base or comparison changes. |
| A vendor result is being used for budget, renewal, or scale language. | The counterfactual, assignment rule, outcome window, leakage checks, and uncertainty. | Attribution, matchback, or model output may describe credit without proving lift. |
| A report or deck is useful but produced by an interested party. | The producer role, sponsor role, outside checks, and limits placed near the claim. | Interest does not make evidence useless, but it raises the standard for method visibility. |
| A case, quote, or example is carrying the broader interpretation. | The population or decision to which the example is being generalized. | One vivid example can illustrate a risk without proving frequency, trend, or cause. |
Run the review in three passes
Pass 1. Name the claim.Rewrite the sentence without promotional, defensive, or headline language. If the wording says caused, lifted, proved, reduced, prevented, or transformed, mark it as causal until the method shows otherwise.
Pass 2. Test the support.Look for the closest source, the denominator, the comparison class, the source role, and the uncertainty statement. If one field is missing, record the specific document, table, method note, or sensitivity check that would reduce doubt.
Pass 3. Bound the next action.Decide what can be repeated now, what needs a caveat, and what must wait for more evidence. The final output should be a sentence a reviewer can defend, not just a color score.
Evidence boundaries before scoring
The Society of Professional Journalists Code of Ethics is a useful editorial boundary: verify before release, use original sources where possible, identify sources and motives, provide relevant source material, and distinguish news from advertising. For public-opinion or survey research, AAPOR's disclosure standards describe the sponsor, instrument, population, recruitment, collection dates, sample, processing, and limitations that make independent review possible. If a commercial message is formatted like editorial content, the FTC's native-advertising guide sets a separate transparency boundary: the commercial nature must be clear and prominent when the format could mislead.
These are source-use boundaries, not endorsements of a vendor, report, or claim. They tell the reviewer what to request and what the evidence can reasonably support; they do not prove that a supplied dataset is accurate, representative, causal, or independent.
Checked-source boundary: SPJ's verification and source-access principles support asking for the closest original record and the source's motives. AAPOR's disclosure standard supports requesting the survey method, sample, precision, processing, response information, and limitations needed for independent review. FTC endorsement guidance supports clear, conspicuous disclosure of an unexpected material connection that could affect how readers weigh a recommendation. These are requests for inspectability; they do not certify a vendor's data or convert an observed difference into lift.
Simulated Worked Example: an observed difference is not lift
Simulated Worked Example - not a real campaign or vendor result. Assume a fictional vendor readout says, "The campaign lifted qualified demand by 20%." Its supplied table shows 300 qualified actions among 3,000 exposed users and 250 qualified actions among a separate 3,000-user comparison group. The readout names the vendor and sponsor and uses a seven-day window, but it does not document randomized assignment, CRM validation of the qualification rule, or whether the groups differed before exposure.
| Review check | Simulated evidence | Score | Decision effect |
|---|---|---|---|
| Observed counts | 300 actions / 3,000 exposed users and 250 / 3,000 comparison users. | 1/1 | The table can support an observed-rate calculation. |
| Denominator | The two group sizes and seven-day window are stated. | 1/1 | The reader can reproduce the rates, subject to the missing eligibility details. |
| Comparison | A separate comparison group is named, but its construction is not shown. | 1/1 | A comparison exists; comparability is still unresolved. |
| Assignment and counterfactual | No randomized assignment, holdout rule, or matching diagnostic is supplied. | 0/1 | Do not convert the difference into causal lift. |
| Outcome validation | The vendor does not show the CRM field, exclusions, duplicates, or qualification audit. | 0/1 | Qualified demand remains a supplied label, not a validated outcome. |
| Source and method disclosure | Vendor, sponsor, window, and table are named; pre-period balance and uncertainty are absent. | 1/1 | Use the result with a narrow descriptive claim only. |
Reproducible arithmetic: the exposed rate is 300 / 3,000 = 10.00%. The comparison rate is 250 / 3,000 = 8.33%. The observed difference is 10.00% - 8.33% = 1.67 percentage points, and the relative difference is 1.67 / 8.33 = 20.0% after rounding. The six-check workflow score is 4 / 6 = 66.7%; that score is a review diagnostic, not a confidence interval or probability of causation.
Decision: hold the claim that the campaign lifted qualified demand. A supportable sentence is: "The supplied table reports 300 qualified actions among 3,000 exposed users (10.00%) versus 250 among a separate 3,000-user comparison group (8.33%), an observed difference of 1.67 percentage points; the materials do not establish incrementality." Re-score only after the assignment rule, pre-period balance, outcome definition, exclusions, and uncertainty are inspectable.
Assumptions and limitations: this simulation treats the two counts and group sizes as internally accurate and assumes both rates use the same seven-day window. It does not test selection, overlap, prior intent, differential exposure, missing outcomes, seasonality, statistical uncertainty, or whether the comparison group is a valid counterfactual. Any of those could materially change the decision.
How to use the result
A green score does not mean the claim is permanently settled. It means the current wording is proportional to the current evidence. A yellow score means the claim may still be useful if the missing piece is named. A red score means the claim should be rewritten, delayed, or treated as a lead for further checking.
Follow-up request map
The review should produce a concrete next request, not a vague concern. Use the weakest row to decide whether the team needs a source document, a denominator, a comparison, a method note, or narrower language.
| If the weak field is | Ask for | Hold back this language until then | Next guide |
|---|---|---|---|
| Source proximity | The original record, full report, data table, transcript, method appendix, or source page behind the summary. | Settled, confirmed, proven, industry-wide, or consensus language. | Source triangulation checklist |
| Denominator or base rate | The eligible universe, starting level, time window, excluded groups, and whether the mix changed. | Large, small, rare, common, growing, shrinking, or record-setting language. | Denominator framing examples |
| Comparison class | The fair baseline, peer group, holdout, matched market, prior period, or expected value. | Better, worse, overperforming, underperforming, or category-leading language. | Campaign baseline comparison checklist |
| Measurement design | The assignment rule, control definition, leakage check, outcome maturity, confidence interval, and sensitivity test. | Drove, caused, lifted, incremental, persuasive, or budget-proof language. | Measurement method selector |
| Claim wording | A revised sentence that names the evidence type, scope, caveat, and decision limit. | Any wording that is stronger than the source, denominator, comparison, or method can support. | Evidence-to-claim language matrix |
Meeting closeout
End the review by assigning the next evidence owner and writing the allowed claim in plain language. If the team cannot name the missing field, the issue is not ready for a confidence score. If the missing field is named but not available, the page, report, or budget note can still move forward with a narrower description.