Source quality
A source quality scorecard for media claims
A link is not the same as evidence. A quoted source, a document excerpt, a chart, and a secondhand summary can all be useful, but they do not carry the same weight. Readers need a quick way to separate strong support from evidence-shaped decoration.
This scorecard is for claims that ask readers to believe something happened, changed, caused harm, improved, failed, or deserves a strong interpretation. It works best before taking a headline personally: identify the claim, inspect the support, then decide how much confidence the support deserves.
The scorecard
Give each dimension 0, 1, or 2 points. A low score does not prove a story is false. It says the public evidence is too thin for a high-confidence judgment.
| Dimension | 2 points | 1 point | 0 points |
|---|---|---|---|
| Proximity | Original record, firsthand data, direct transcript, or full document. | Reported summary with enough detail to locate the original basis. | Secondhand description with no path back to the underlying record. |
| Completeness | Enough context to see what was included, excluded, and compared. | Some context, but key definitions or exclusions remain unclear. | Selective excerpt, isolated quote, or chart without denominator. |
| Specificity | Named claim, time period, population, metric, and comparison class. | Most details present, but one major boundary is missing. | Broad language such as "many," "experts," or "critics" does most of the work. |
| Independence | Support comes from a source with visible methods and no direct stake in the outcome. | Interested source, but the article names the interest and adds outside context. | Interested source supplies both the claim and the interpretation. |
| Uncertainty | The story states what is known, unknown, disputed, and still being checked. | Some caveats appear after the main claim has already been framed strongly. | Uncertainty is hidden behind confident language. |
| Replicability | A reader can follow the evidence trail and test the central comparison. | The trail is partly visible, but one step depends on trust. | The claim cannot be inspected without private access or unnamed interpretation. |
How to read the total
10-12 pointsStrong public support. Readers can inspect the main evidence, understand the comparison, and see the limits of the claim.
6-9 pointsUsable but incomplete. The article may be directionally useful, but the strongest interpretation should stay provisional.
0-5 pointsWeak public support. Treat the story as a prompt for further checking, not as a settled account.
Turn the score into claim language
The total is useful only if it changes the sentence a reader is willing to repeat. A high score can support direct language. A middle score should keep the claim conditional. A low score should move the reader into a source-checking task.
| Score pattern | Acceptable reader language | Do not say yet | Next check |
|---|---|---|---|
| High total, no zero-point dimension | "The public record supports this claim, with stated limits." | "This proves the strongest interpretation." | Check whether the comparison class matches the headline frame. |
| High total, one dimension at zero | "Most support is visible, but one missing boundary keeps confidence capped." | "The evidence is complete." | Find the missing denominator, counter-source, method note, or original record. |
| Middle total with interested-source dependence | "The claim is plausible but still relies on a motivated source." | "Independent evidence confirms it." | Look for method-bearing analysis or a primary record outside the interested party. |
| Middle total with weak uncertainty language | "The direction may be useful, but the strength and scope are unsettled." | "The size of the effect is established." | Check sample limits, confidence ranges, missing definitions, and alternative explanations. |
| Low total or no inspectable trail | "This is a lead for more reporting, not a conclusion." | "The story shows what happened." | Return to the source trail and identify what would make the claim inspectable. |
Source tiers
Strong stories often combine several tiers. Weak stories ask a low tier to do work that only a higher tier can support.
| Tier | Examples | Best use | Main risk |
|---|---|---|---|
| Primary record | Filings, transcripts, full reports, public data tables, direct statements. | Establishing what was said, reported, counted, or formally decided. | Readers still need context and definitions. |
| Method-bearing analysis | Research with disclosed sample, measures, limitations, and uncertainty. | Estimating change, association, prevalence, or effect size. | Methods may not match the headline claim. |
| Expert interpretation | Credentialed analysis tied to a visible field of expertise. | Explaining mechanisms, tradeoffs, and uncertainty. | Authority can substitute for evidence if the basis is not shown. |
| Interested-party statement | Company, campaign, advocacy, plaintiff, defendant, or vendor position. | Representing what a stakeholder says or contests. | The source has a reason to select favorable facts. |
| Secondhand summary | Aggregated writeups, paraphrases, newsletters, and social posts. | Discovery and orientation. | The frame can drift away from the original record. |
Fast red flags
- The headline makes a causal claim while the source only shows timing or correlation.
- A single anecdote carries a population-level conclusion.
- The article reports a percentage change without the starting level.
- One side is described through documents while the other is reduced to a paraphrase.
- Anonymous sourcing is used for interpretation rather than narrow factual context.
- The strongest caveat appears only after the reader has been given a confident frame.
Useful rewrite questions
| If the story says | Ask this before trusting it |
|---|---|
| "A growing number" | Growing from what baseline, over what period, and compared with what peer group? |
| "Experts warn" | Which experts, in which field, with what evidence, and what do comparable experts dispute? |
| "Data shows" | Who collected the data, what is the denominator, and what was excluded? |
| "Critics say" | Are critics providing facts, interpretation, incentives, or only a reaction quote? |
| "The move sparked backlash" | How large is the backlash relative to the affected population or normal response level? |
Worked scoring pass
Suppose a story says a new policy "sparked a surge" because one organization shared a chart and several observers reacted. The chart is useful, but the score should stay restrained until the source trail can support the causal frame.
| Dimension | Score | Why it earns that score | How the reader should soften the claim |
|---|---|---|---|
| Proximity | 1 | The chart is visible, but the underlying table and collection rule are not linked. | Say "reported chart" rather than "data proves." |
| Completeness | 0 | The starting level, affected population, and exclusions are missing. | Do not repeat "surge" without a denominator. |
| Specificity | 1 | The time period is named, but the comparison period is vague. | Use "during the reported window" instead of a broad trend claim. |
| Independence | 0 | The organization supplying the chart benefits from the interpretation and no outside record is cited. | Call it an interested-source claim until another source can inspect it. |
| Uncertainty | 1 | The story mentions other possible explanations but only after the confident frame. | Keep the causal language conditional. |
| Replicability | 0 | A reader cannot reproduce the central comparison from the public material. | Treat the story as a prompt for follow-up reporting. |
The total is 3 out of 12, so the careful reader should not say the policy caused a surge. A stronger version would be: "One interested source reported a short-window increase, but the public record does not yet show the denominator, comparison period, or independent confirmation."
Takeaway
The reader's job is not to demand impossible certainty. It is to keep confidence proportional to the evidence. A serious article makes that easier by showing the record, defining the comparison, and separating what is known from what is inferred.
Move from source scoring into the evidence trail.
Use these routes when a source score raises a question about independent confirmation, the fuller evaluation desk, reusable reference records, or the broader media-bias frame.