Surveys and polling: advice from pros
- When reporting poll results, avoid using decimal points or tenths of a percent—that is, report 28%, not 28.4%. (The margin of error will always be at least 1 percentage point, so tenths of a percent are effectively meaningless and misleading, suggesting that the results are more accurate than they actually are.)
- Don’t place too much weight on any one poll. It’s best to compare several similar polls.
- Neither should you presume that an aggregate of smaller polls necessarily adds to accuracy or precision. Some aggregators use more sophisticated methods than others, and the quality of their results can vary greatly.
- Don’t forget that even small differences in question order and choice of words can significantly alter results. (For example, asking survey participants about “euthanasia” versus “physician-assisted death”). For some topics, consider directly quoting the question in full, so readers can see how it was asked.
- Remember that all poll estimates are inherently uncertain. Margins of error are typically calculated at a “95% confidence level,” which means that in about 5% of poll results — that is, five results out of 100 — the truth of what’s happening in the population will lie outside the margin of error’s bounds.
- Don’t assume that a poll with a large sample size has high statistical accuracy. The increased statistical precision achievable with large samples can be overwhelmed by the uncertainty and bias potentially introduced from such factors as poorly designed survey questions, flawed data collection, and improper statistical analysis.
- This is especially important when looking at online polls, where large numbers of respondents can be amassed cheaply. At very high levels—above 1,000 or so respondents—how many individuals were selected is less important than how they were selected and their responses analyzed.
- Note that terms like “nationally representative,” “organic sampling,” “next-generation sampling,” “representative of all U.S. adults,” and “random sample” may be defined differently by different pollsters. It’s best to ask precisely what is meant in each case. Also note that for election polling, the distinction between “likely voter” and “registered voter” may be especially important.
- Even well-designed Presidential election polls can lead you astray. One thing to watch for: nationwide polls of voters are usually designed to capture only the popular vote, not electoral college outcomes. And while poll aggregators often build electoral college weights into their models, they rely on state polls, which tend to be less well funded, smaller, and less precisely weighted than national polls. Late-deciding voters can also significantly swing elections away from poll predictions.
- A final, important point: Surveys are done on a vast range of topics other than electoral preferences. They provide essential data on economic activity, health status, drug use, consumer behavior, and countless other measures that are critical to responsible, democratic policy making and the intelligent allocation of resources. In many of these domains, surveys are quite good at predicting behaviors and needs. When reporting on polls and surveys, treat each with the same fairness you demand from them!
Excerpted from Surveys and Polling, compiled by SciLine and the American Statistical Association.
