Survey evidence
Survey and poll claim checklist
Organization byline: Measurement Press editorial desk. This page does not list an individual author or reviewer. Read the editorial method and corrections policy.
About 8 min read
A survey can be useful evidence, but the headline often strips away the method. Before trusting a poll, brand study, customer survey, or reader research claim, check who was asked, how they were reached, what wording they saw, when they answered, and how much uncertainty remains.
This page is for media readers, editors, analysts, and marketers who need to keep survey evidence inside its limits. It does not assume that every poll is weak. It asks whether the disclosure is strong enough for the claim being made.
One-page source check
| Question | Strong disclosure shows | Weakens confidence |
|---|---|---|
| Who sponsored and conducted it? | The sponsor, field organization, funding source, and any material interest are visible near the result. | The story says "a new poll finds" without naming who paid for or ran the work. |
| Who could be included? | The target population, geography, language, eligibility rules, and sample frame are clear. | A member panel, customer list, newsletter audience, or opt-in sample is framed as the general public. |
| How were people reached? | The mode and recruitment path are named: phone, address-based sample, probability panel, online panel, intercept, customer list, or mixed mode. | The source gives only the number of respondents and hides how they entered the survey. |
| When was it fielded? | Field dates are shown and interpreted alongside major events, campaign bursts, price changes, news cycles, or product changes. | The headline treats the result as current even though the field window is old or event-sensitive. |
| What exactly was asked? | Exact question wording, answer choices, order, screening rules, and topline results are available. | A paraphrase, chart title, or press summary replaces the actual question. |
| How was it weighted? | Weighting variables, subgroup bases, design effect or equivalent precision limits, and unweighted counts are disclosed when relevant. | Small subgroup differences are promoted without showing subgroup size or weighting sensitivity. |
| How uncertain is it? | Sample size, margin or interval, subgroup uncertainty, and non-sampling caveats are stated close to the claim. | A one- or two-point difference is written as a real lead, trend, or shift without uncertainty context. |
| What is the comparison? | The story compares the result with prior waves using the same method and wording, or clearly labels a looser comparison. | The article compares unlike polls, different modes, changed wording, or different populations as if they were identical. |
Read results in order
Evidence boundary for the checklist
AAPOR's disclosure standards treat sponsorship, population, recruitment, mode, field dates, sample size and precision, weighting, processing, and limitations as information readers should be able to inspect. Pew Research Center's survey-methodology overview places coverage, sampling, nonresponse, measurement, and processing or adjustment error inside the total-survey-error frame and points readers to exact question wording.
This page uses those references as an inspection boundary, not as a quality seal. A disclosed method can still be poorly matched to the population or claim, and a margin-of-error calculation does not repair coverage, nonresponse, wording, timing, or other non-sampling limitations.
What the verified sources establish
AAPOR's public-results standard calls for enough method detail to permit independent review, including the sponsor, instrument wording, target population, recruitment, mode, field dates, sample size and precision, weighting, processing, and design limitations. That is a disclosure test, not a finding that the survey is representative or persuasive.
Pew's total-survey-error frame separates coverage, sampling, nonresponse, measurement, and processing or adjustment error. The practical consequence is that a reported margin of error addresses only part of the uncertainty; it cannot by itself settle wording, coverage, recruitment, or population-fit questions.
1. Start with population"Adults," "registered voters," "likely buyers," "customers," "readers," and "respondents who saw an ad" are different populations. A result can be valid for one group and misleading when promoted to another.
2. Read the exact wordingSmall wording changes can alter answers, especially when the topic is unfamiliar, emotional, technical, or tied to social approval. Do not rely on the chart label when the question text is available.
3. Check timing before trend languageA poll fielded after a major event may measure immediate salience more than durable opinion. A brand survey fielded during a promotion may measure campaign context and offer context together.
4. Treat subgroup claims as fragile until bases are visibleSubgroups often carry the most interesting headline and the weakest precision. Ask for unweighted base sizes, intervals, and whether the subgroup was planned before results were inspected.
5. Separate survey movement from causal impactA survey can show what respondents reported in a window. It does not prove what caused the answer unless the design preserves a credible comparison.
Simulated worked example: does a five-point movement ship?
This is a fictional worked example, not a reported poll. Use it to reproduce the arithmetic and see why a large-looking headline can still require a hold. Suppose a sponsor reports that support increased from 44% to 49% in two independent waves of 1,200 respondents each. The current wave contains 588 support responses (588 / 1,200 = 49.0%); the prior wave contains 528 (528 / 1,200 = 44.0%). The observed movement is therefore 49.0% - 44.0% = +5.0 percentage points.
| Disclosure gate | Fictional packet | Result |
|---|---|---|
| Sponsor and field organization | Both are named, with the funding relationship stated. | Pass |
| Population and recruitment | Target population, panel source, eligibility, and recruitment path are shown. | Pass |
| Question wording and order | The same question and answer choices are shown for both waves. | Pass |
| Field dates and event context | The current wave ran immediately after a major announcement; the prior wave did not. | Hold |
| Weighting and precision | Weights are disclosed; an illustrative design effect of 1.5 is supplied. | Pass |
| Disposition and nonresponse limits | Completed interviews are shown, but sample dispositions and nonresponse information are missing. | Hold |
For a reproducible scale check, a simple-random-sample 95% margin-of-error approximation at n = 1,200 is 1.96 x sqrt(0.25 / 1,200) x 100 = 2.83 points. Applying the fictional design effect gives 2.83 x sqrt(1.5) = 3.47 points per wave. An illustrative independent-wave difference interval is about 1.96 x sqrt[(0.49 x 0.51 / 1,200) + (0.44 x 0.56 / 1,200)] x 100 = 3.99 points. The five-point movement is larger than that arithmetic alone, but the calculation does not repair an event-timed field window, missing dispositions, coverage gaps, nonresponse, wording effects outside the supplied packet, or other total-survey-error sources.
Decision: hold the durable-trend headline. A careful interim line could say that the fictional current wave measured 49% support, five points above the prior wave, while the field timing and missing disposition information limit interpretation. Release a stronger "support increased" claim only after the method packet makes the two waves comparable and the remaining limitations are visible.
Reproduction note: replace the fictional counts, sample sizes, field dates, wording, weights, and design effect with the source report's values. If the waves are not independent, if the sample is non-probability, or if the report does not justify a margin-of-error calculation, do not reuse the interval arithmetic as if it were a test.
Poll-to-headline translation
| If the report says | Careful wording | Overclaim to avoid |
|---|---|---|
| A candidate leads 48% to 46% with a three-point margin. | The race is close in this poll; the measured difference is not enough for a confident lead claim. | The candidate is ahead. |
| Support rose from 31% to 36% using the same method and question. | The poll shows a possible five-point increase; check uncertainty and field dates before calling it a durable shift. | Opinion surged. |
| Customers who answered a survey report high satisfaction. | Responding customers report high satisfaction; nonresponse and customer-list coverage remain important limits. | Customers are satisfied. |
| A small subgroup differs sharply from the full sample. | The subgroup result is directional until the base size, interval, and preplanned comparison are visible. | The subgroup has a clear separate view. |
| Exposed respondents remember the ad more often. | The study shows higher reported recall among the measured exposed group in this window. | The campaign changed the market. |
Brand and reader research crossover
Brand lift studies, reader surveys, customer panels, and public opinion polls share the same disclosure problem: a persuasive summary can hide the population, question wording, recruitment path, weighting, and uncertainty. For advertising measurement, add one more layer: exposed and control respondents must be comparable before survey movement can be treated as campaign evidence.
Use survey evidence for the decision it can support. It can guide message testing, audience understanding, editorial framing, or a stronger experiment. It should not become proof of sales lift, public consensus, or long-term brand change unless the design and comparison can carry that claim.
Minimum citation packet
- Sponsor, field organization, funding source, and disclosure of material interests.
- Target population, geography, eligibility rules, field dates, language, and survey mode.
- Sample source, recruitment path, completed interviews, subgroup bases, and weighting method.
- Exact question wording, answer options, order effects that matter, and topline results.
- Margin or interval for full-sample and subgroup claims, plus caveats for non-sampling error.
- Comparison standard: same question over time, benchmark data, exposed/control balance, or a clearly labeled loose comparison.
Official references
AAPOR disclosure standards identify the sponsor, instrument wording, population, recruitment, mode, field dates, sample size and precision, weighting, processing, and limitations as method details for public results. The AAPOR Transparency Initiative is a disclosure framework, not a judgment that a disclosed method is rigorous. Pew Research Center's U.S. survey methodology explains total survey error and publishes exact question wording in survey toplines. These sources set the boundary for this checklist; they do not validate any fictional example or any study reviewed here.
Takeaway
The question is not whether a poll is persuasive. The question is whether the method is inspectable enough for the headline. Strong survey writing keeps population, wording, timing, weighting, uncertainty, and comparison close to the claim.