Reporting on opinion polls and surveys

[00:00:46]

RICK WEISS: Hello, everyone, and welcome to SciLine’s media briefing on reporting on opinion polls and surveys. I’m SciLine’s director, Rick Weiss. And for those not familiar with us SciLine is a philanthropically funded, editorially independent free service for journalists, based at the nonprofit American Association for the Advancement of Science. Our mission is simply to make it as easy as possible for you, as reporters, to include scientists sources, and scientifically-validated information in your news stories, whether those stories are about science or about other things on your beat that would benefit from some scientific context. And really, I’ll just say for this briefing, there is no better example of stories that are not about science, but would benefit from having some science, than political stories. The issues that candidates are arguing about every day now, like immigration, crime and safety, voting integrity and security, are not just “he said, she said” matters of opinion. They are things that actually scientists, often social scientists have been studying for years, and about which there are data that you could include in those stories. So please, as you write about these issues in the coming months, consider coming to sciline.org. Check out, among other things, our expert matching service, which can provide you with an expert, a scientist, who studies the topic you’re writing about on deadline, and that will help you get some of that rigorous data into those stories that are otherwise just seem to be based on opinion or anecdote.

A couple of quick logistical details before we get started today. We have three panelists who are going to talk for about seven minutes each, before we open it up for Q&A. To enter a question, just go down to the bottom of your Zoom screen to that Q&A icon. Hover over that, click on it, and let us know who you are, who’s your outlet, and what is your question. If you want to direct it to someone go ahead and mention that as well. A video of this will be available very soon after it ends in an hour. But if you need some raw copy sooner, just get in touch with us through that Q&A icon during the briefing, and let us know and we’ll send you something as soon as it’s over. The transcript will be up a day or so after the video.

I’m not going to do full bio introductions. There are bios of our speakers on the website. But I will just say that we will hear first from Courtney Kennedy. Courtney is vice president of methods and Innovation at Pew Research Center. She oversees Pew’s American Trends Panel, and conducts experimental research to improve the accuracy of public opinion polls. Next, we’re going to hear from Gary Langer, who is president and founder of Langer Research Associates. That’s the company that produces the ABC News/Washington Post polls for ABC. And he also manages international surveys for the Pew Research Center. He also, like me, is a recovering reporter, including a 10-year stint at the Associated Press. But unlike me, he has a couple of Emmys sitting on his mantle. And we can get into that in the Q&A if we’re desperate for things to talk about. And third, we’ll hear from Amelia Thomson-DeVeaux, who is the polling editor at the Associated Press. And before that was a senior reporter at FiveThirtyEight, where she covered politics and elections. So let’s get started. And over to you, Courtney.

[00:04:13]

COURTNEY KENNEDY: All right, thank you. OK. It’s great to be with you all today. I’m going to start by talking briefly about the landscape of polling that we find ourselves in this year. And then I’ll mention some really specific things to watch out for when covering polls. Okay. So we are in a period where there is just tremendous diversity in how polling is done, for better or worse. That’s how it is. And that’s what this first chart here is showing us. So along the bottom there, along the x-axis is time, right, so starting back in the year 2000, and then looking all the way till roughly today. You can see that 20+ years ago, almost all pollsters were using live telephone to do their polls. That’s represented in the dark blue. And over time, you see all these new colors come in. Those colors represent different polling methodologies, different ways that pollsters sample people or interview them. And you can see sort of in the early 2000s, right, you’ve got this huge migration of polling online. And today, there are different flavors of online polls, we could really get into details about. There’s also hybrid polls. A lot of pollsters today will interview some people online, others by text, or maybe online and phone. There’s a lot of combinations that you can see today. And the other main point, I would highlight here, is you just see the growth in the field itself, right? So sort of the, the y-axis is the volume of polls over time, and in that particular type of surveying. And so you can see that the field has grown. And one thing that’s key here, it’s similar to I suspect your experience in the news industry, which is that, over this period of time, the barriers to entry in the profession have really disappeared, right, and barriers to entry to be a journalist have largely gone away. The same with polling, because now there are many, many websites you can go to. And if you have a few thousand dollars, you can, quote/unquote, go do a national poll and write up a press release. So that’s something to really be aware of as reporter, and it really does put an obligation on reporters to figure out well, who did the poll, and do they have a track record? Are they established? Do they really have credibility as a pollster?

OK, now, I’d like to walk you through one specific example of what polling can look like. This example is from state of Florida, the presidential race in 2000. I pulled this graph from a pretty popular polling aggregator website. And I think a normal person would probably look at these data. And you have the conclusion that, wow, Biden has a consistent lead, a pretty strong lead in this race. He looks like he’s ahead in Florida. There’s several problems with this, though. One is that the chart itself is zoomed in extremely tightly. And what that does is it makes very small changes, less than even a percentage point, makes those small changes look meaningful, significant, when they’re really not. They’re probably just noise. We also see that there’s decimal places shown for the candidate support. And as much as I love polling, I really don’t like it when decimal places are shown. Because the polls are great, but they have limits. And polls are just not a precise enough tool to get it right in the decimal place. That is just not at all a reasonable expectation to have about polling. And so I think that when it’s written up, and we show a decimal place, that it’s very unhelpful to sort of signal that precision, when it’s really not there.

And finally, as I said, I think a normal person would say, like, “Oh, there’s a lead in this race.” Right? “It looks like one candidate is ahead.” But what you should keep in mind is that, for example, I’m showing state polls here. In the last two, president—presidential election cycles, the state polls have been off by an average of five percentage points. What we have here is less than one percentage point. And so anytime I see a poll of two candidates, and the candidates are within five points of each other, I interpret that as—that polls basically tied. Because, historically recent history, that’s about all you can expect out of state level polling. So if it’s within that five-point margin. I think it’s safest to just know that poll is telling you it’s a competitive race. And to drive home this point, I took this exact same data, and I just charted it, in I think a little bit more a less sensationalized way. You let the access go all the way from zero to 100. And it’s clear here, that this is just a very competitive race. And the polls are not going to be able to tell you, with any confidence, which candidate might be ahead. It’s way too close.

Ok. Another thing that folks new to reporting on polls might not realize, is that sometimes the campaigns themselves, the campaign pollsters, will selectively release polling results if it’s good for their campaign. There’s an example of this a couple years ago in South Carolina Senate race. You had Lindsey Graham running for reelection. But his Democratic challenger’s polling firm selectively released this one poll that had the Democrat looking like they were doing very well. And it kind of spun up this mini news cycle about that being a competitive race. But really, that wasn’t reality. Right? That was one selectively released poll by a campaign pollster.

And so just be aware that this kind of thing can happen. And it’s not really a credible poll, because you’re not seeing the other 95 polls that pollster did that didn’t look anything like that. A kind of similar thing to watch out for is polling organizations that all of a sudden—all of a sudden appear out of nowhere. So in a Michigan Senate race a few years ago, you had Debbie Stabenow running for reelection. And there was a poll that was picked up by a few outlets, that claimed that Kid Rock, the musician, that Kid Rock was leading the poll in the Michigan Senate race. And the thing to watch out for here is that if you really dug under the hood about, okay, who’s this pollster, do they have a track record, you’d see that this polling company sort of emerged out of thin air miraculously weeks before this very sensational headline came out. And so it’s really not even clear, to this day, if this was a real poll, or what was going on. So you really want to make sure, again, that any poll that you might pick up is established, they have a track record, they know they know what they’re doing. Okay. And with that, I think I’ll turn it over to my colleagues.

[00:11:24]

RICK WEISS: Thank you, Courtney. Great introduction. Some great examples. And over to you, Gary.

[00:11:38]

GARY LANGER: You guys got my slides?

[00:11:39]

RICK WEISS: Yeah.

[00:11:42]

GARY LANGER: Terrific! All right. Polling primer in five or so points. Great job, Courtney. And thank you, Rick. And SciLine does a terrific service to working reporters around the country. And I just really want to appreciate you for that, and for the opportunity to participate today.

Point one about election polling is that forecasting isn’t easy. This is the Hurricane Ian in 2022. Forty-eight hours before the hurricane made landfall in Fort Myers on September 28th, the forecasting model predicted it would make landfall in Tampa on September 29th. This is not to throw shade on hurricane forecasting, but just to make the point that this is tricky business. Look at all the squiggly lines trying to estimate where this thing was going to land. There’s a lot of complexities involved in forecasting hurricanes, and elections as well.

Second point is that in pre-election polls, forecasting isn’t the point. As Courtney just showed, a very close race is a close race. And how boring is it to recite it, endlessly, that it’s a close race? Look at all the things that we don’t know about elections with the—in the absence of high quality survey research. What issues motivate likely voters? Which don’t? How do people come to their decision, whether they’re going to vote or not? What policy preferences matter? What candidate attributes? How do campaign controversies influence the contest, if at all? How and why, again, likely voters are coming to their choices whether to participate? And if so, whom to support? What does the election mean? At the end of the day, we will know who won. We don’t need polls to tell us that. But what we won’t know, without the contribution of polls, is how and why. And it’s a really important element. We need to cover not just the campaign, but this broader thing: the election. Remember that cone of uncertainty in hurricane forecasting, other polls sample unknown population. You sample them, you get your results, you’re good to go. Pre-election polls have to estimate, and then sample an unknown population: would-be voters. And this layer of estimation produces additional uncertainty. Now, that uncertainty is exacerbated by externalities, including, for example, holding an election in the midst of a fundamental change in how people vote, with this huge shift to early and absentee voting, we’ve experienced, in a time of extraordinary political emotion, with a candidate facing 91 felony counts, with another candidate who seems really old to a lot of people, after a 40-year high.

There’s all sorts of factors that go into the dynamics of an election, that make estimation more and more challenging. Not to say it can’t work. These are ABC News/Washington Post national surveys, final pre-election estimates for the national vote, and it’s a very long and strong track record of accuracy. We didn’t do one in 2020. We did some state polls, and state polls are trickier, as also Courtney mentioned. We can discuss why that is in the Q&A. Not all polls are created equal. There’s this issue of probability versus convenience sampling, which I’m going to return to in just a minute. And then we have good versus poor practices in sampling, and questionnaire design the forgotten stepchild of survey research, in weighting and analysis. How do you pick through it? You have to insist on transparency. We need full disclosure of all the methods. It’s not just numbers and percentage signs. We need to know how these numbers were obtained. Reporters, too often, want to grab the numbers and run. And I like to say that running with data is like running with scissors. It’s really easy to get hurt.

The key point about this point is that reporting polls requires reporting. And your organization, your news organization, requires standards, that I can’t get every survey researcher to agree on what are, or are not, valid and reliable methods. But where we do agree, is that it’s important to make our best effort to understand it, to develop and enunciate standards, and be in a position to explain and defend them. Every newsroom needs to do this. What kind of polls are you going to report? What not? Most importantly, how, why? How you come to these determinations? Probability versus convenience sampling. I’m going to talk about this real quick. There will not be a quiz. The guiding principle of inferential statistics is that in order to make inferences about a full set, we example—we examine a randomly selected subset. Every unit in the full set needs a known nonzero probability of being selected. That’s why it’s called “a probability sample”. Pure probability is a tall order, but with probability based sampling, with efforts to minimize departures from pure probability, it’s proven to be really highly robust. We can get good estimates. Convenient sampling, on the other hand, abandons the theoretical framework of inferential statistics. It relies on nonrandom purposeful selection and/or self-selection.

The most prominent are often online surveys conducted among people who sign themselves up to click through questionnaires on the Internet in exchange for cash and gifts. Much research shows non-ignorable inconsistencies within one of these panels across time, across different panels within a single point in time, and compared to population benchmarks. The AAPOR, the American Association for Public Opinion Research, did a report on these online panels. This was in 2010. It’s still up there. I don’t believe anything’s changed. Researchers should avoid these panels when one of the research objectives is to accurately estimate population values, which seems to me, the purpose of the enterprise. The nonprobability nature of these panels violates the underlying principles of probability theory. There currently is no generally-accepted theoretical basis to which—from which the claim that surveys using samples from non-probability on life panels are projectable to the general population, so claims of representative representativeness should be avoided. They go onto say that participants in these panels are really different from everybody else. And that reporting a margin of sampling error for these sorts of panels is fiction.

Back in 2006, The New York Times had its polling standards. In sampling, they said, “We need a probability sample.” In Internet and opt-in polls, they said, “These do not meet the Times’s standards, regardless of the number of people who participate. In order to be worthy of publication of the Times, a survey must be representative that is based on a random sample of respondents. A survey that relies on web respondents who have selected themselves,” it says here, “it’s not reliable”. OK. Now, let’s put some of this in practice. The Associated Press has a survey it calls the Votecast, “a modern approach to election surveying,” it says. Up until early last week, when you Googled “AP Votecast”, you’d see this page, in which they said, “We used a probability based state-by-state survey.” This is great. It’s probability-based. That’s the gold standard of valid and reliable survey research. However, when I was looking around, I found another description of AP Votecast from another source, partner of theirs. It says they take interviews with a random sample of registered voters and combine them with interviews selected—from individuals selected using non-probability approaches. So over here, AP is saying it’s a probability-based survey. And over here, we’re seeing something like maybe not. I shared my slides with Rick, who shared him with Amelia and the AP. He ratted me out. And they changed their website. If you go to this site today, the website says this: “We use a probability-based survey. We add to that a survey of self-identified registered voters using online panels.” Maybe a little better. AP is still not fessing up that they use non-probability data. I’m not sure why. We can have endless debates about whether or not it’s appropriate to meld probability and non-probability data. Some people, maybe me, think it’s sort of like averaging champagne and turpentine. If it’s your thing, drink up. Others may find it defensible. But what it needs to be is disclosed, and full disclosure as essential, and that’s where the reporting piece comes in.

I’m going to give you one other real quick. This came — landed in my mailbox the other day. American Enterprise Institute is having a “What the Hell is Going On” event, where they have some smart people talking about what’s happening in the news. Their come-on said: “President Joe Biden is the least-popular president in the history of presidential polling. Former President Donald … ” —I said because I haven’t spent some time with these sorts of things. Wait a minute. President Joe Biden is the least-popular president in the history of presidential polling? The beautiful thing about survey research is we can check this stuff out. A quick visit to Gallup, sorry about that, will tell us that Biden’s current rating, when this was written, was above his own low. And there was not one, not two, but six, previous presidents since Harry Truman, who had lower, in many cases substantially lower, approval ratings. So even the American Enterprise Institute, a research institute and respected the organization, can resort to bloviating and misrepresenting survey data. And as reporters, we therefore need, not only in cases of little known or unknown producers, but in cases of everyone to check it out.

Now, all that said, just to return to election work. Pre-election polling, as I’ve said, like forecasting hurricanes, does provide essential if sometimes imprecise information. Campaigns and interest groups conduct their own polls to try to manipulate public attitudes and behavior, and to manipulate media coverage of issues and candidates in order to achieve their goals. In the absence of quality, independent public interest polling, we would be defenseless to this manipulation. That’s a fundamental reason we do it. Thank you very much.

[00:21:19]

RICK WEISS: Thank you, Gary. And a reminder to reporters, all these slides will be available immediately after the briefing. So if you want to go back and check out some of those details, it’ll be easy to look at those. And over to you, Amelia.

[00:21:32]

AMELIA THOMSON-DEVEAUX: Great. All right, let me go ahead and share my screen.All right. Can everyone see?

[00:21:47]

RICK WEISS: Perfect!

[00:21:48]

AMELIA THOMSON-DEVEAUX: Great! So I’m going to be talking about rules of thumb for writing about surveys, both as now a polling editor, but also a reporter who used to write about surveys all the time, and particularly in an election year, because we’re coming up to the presidential election. And that is what a lot of the polling that we’re seeing is about. So just some questions to have in mind when you’re evaluating whether a poll is trustworthy. Who paid for the poll? Does the organization sponsoring it have some interest in having a particular outcome from the poll? Who was interviewed? Are we looking at U.S. adults? Is this a poll of retirees? Is it another group? How are they interviewed? This is — goes back to the discussion of methods that Gary and Courtney have already outlined. Which questions were asked, and in what order? When was the poll conducted? You shouldn’t assume that just because it was just released, that it was just conducted? And what is the margin of error, which gives you a sense of the uncertainty. And when writing about horse race polls, there are some guidelines, I think, that that can be helpful for reporters.

So we’re talking here about polls that are asking whether people would vote for, in this case, Trump or Biden, in the November election. Take early polls with a grain of salt, and I would say now is still early. It may feel to us like the election is very close, but to the majority of Americans, they are not thinking about the election, the way that journalists are. It’s still very far away. And a lot can change between now and the election. You also want to look at the margin of error before assessing the state of the race. And be really careful when you’re talking about something like a lead. Polls have margins of error. Elections don’t. But it’s tempting to, sometimes, think about polls in the same way that you would think about an election. So let’s say you see a poll that has Trump at 47%, has Biden at 40%. That might look like a huge margin, like a real blowout. But if the poll has a 4 percentage point margin of error, actually, it could be quite a bit closer. And you want to be really careful about when you say that a candidate has an actual lead, to bear that margin of error in mind. And with subgroups, in particular, the margin of error can be even higher.

So if you’re comparing men and women, you want to just make sure that you have a sample that’s big enough to tell you something statistically-meaningful about the difference between those groups. It’s also important to be conscious of what polls can and can’t tell you. Polls are not predictive. There’s uncertainty. And even they’re — even when they’re reflecting the reality of right now, things can change in ways that are predictable. I was just asked the other day about some Maryland Senate polls that were showing Larry Hogan with a big lead over the Democratic candidates. And that looks good for Hogan. Right? But one thing you see also in the poll is that Hogan has much higher name recognition than either of the Democratic candidates, and also that the voters prefer a Democratic Senate by a very large margin.

So this suggests that even though this poll—we don’t have reason to think this poll is wrong, or telling us something really untrue, but once there is a Democratic candidate, who has higher name recognition, things could shift. And you can always include context. You want to ask: what question am I trying to answer? Am I trying to help people understand how are they feeling about the candidates? Well, maybe you can look at Trump or Biden favorability, in addition to the horse race number. Perhaps, you can look at Biden approval on issues. There are lots of other polls that can help shed light on the why. And as Gary said, it’s often much more interesting to look at those than to repeat over and over again, that this is a competitive race. I think one resource that AP uses, as Gary mentioned, in trying to understand the how and why of elections, is AP Votecast. This is a large survey that helps us explain the election. So it helps us understand the composition of the electorate, and what people were thinking about as they went to vote. And the reason that AP uses this survey, instead of something like a traditional exit poll, is that the way people vote has changed dramatically. You can no longer stand outside a polling place, and survey people about why—who they voted for and why. Many Americans now vote by mail, or vote early. So in 2018, AP, with partners, pivoted to a large survey that starts in the week leading up to Election Day. It concludes as polls closed. It includes demographic questions and vote choice questions, but also a lot of questions about issues that really helped get at voters’ potential motivations. And it is a probability-based state-by-state survey, combined with a large opt-in sample, that allows for a very large number of interviews, while still maintaining methodological rigor. And the survey really is an example of how you can, you can report on vote choice, but you can also tell the story of the why, and the how.

And in writing about polls, I’ll just talk quickly about a few more common pitfalls. You’re science reporters, use your critical eye, when talking about the data. Question order really can matter a lot. This is something I discovered a long time ago, but way before I was working at the AP, when I was working on a survey about marijuana. We started with a question that had something to do with marijuana use around children. And then later in the survey, we had a question about whether people had used marijuana at all. We got the data back, that second question was really weird. The marijuana use number, it looked really, really low. And our hypothesis was that we had introduced this idea of maybe — maybe marijuana is not great, asking about it in the context of kids. And that made people less willing to admit that they had used it. So it really can matter. You want to make sure that two polls can be compared, that they use similar methodology, that they’re looking at the same population. You don’t want to compare polls of U.S. adults versus registered voters. For example, that’s not the same population. Pay attention to how the data is visualized. You don’t want to imply, for example, in one of those charts, like Courtney was showing, where we’re seeing what a polling — what polling numbers have looked like. You don’t want to, for example, put arrows at the end, suggesting that they’ll continue in a particular trajectory. Not that Courtney would ever do that, but I have seen it done. And again, look for context. Share that context with readers. Often in poll top line questionnaires, which are the documents that will outline the results of each question, they’ll include trends, if the trend question has been asked before. You can include several surveys that ask about something in different ways and highlight different sides of an issue. Really, your job, as a reporter, is to try to gather multiple sources of evidence, and help shed light on what Americans think. And that’s a complex thing that one poll question, or one survey, is not going to answer particularly in an election year like this one. So I will — here are a few resources. So I will turn it back over to you, Rick.

[00:30:02]

RICK WEISS: Fantastic. So much wisdom already in these presentations, I really appreciate all the examples and categories of all the ways that reporters can go wrong. It’s almost like there’s only a few ways to get it right, but that’s worth striving for. So I’m going to remind reporters, as we get started in the Q&A here, that if you want to ask a question, please go down to that Q&A icon at the bottom of your screen, and go ahead and start entering those. But meanwhile, I like to always take the privilege of asking the first question during these media briefings, and it’s typically pretty much the same question. And it’s asking all of our three experts here to really put their news consumer hats on, and as experts, who are themselves readers and watchers of the news, to tell you reporters, something about what they either really appreciate, or don’t appreciate about the way a lot of reporters handle this topic of covering polls and surveys. So why don’t we take a couple of minutes on that just to get started? And I’ll start with you, Courtney.

[00:31:03]

COURTNEY KENNEDY: Sure. I was reading a poll article, a week or two ago, I saw something I loved. The article, it was — it was focused on a poll, but they had an explicit box, where they said, “Why are we reporting on an election poll, and it’s only April. It’s really early in the campaign.” So they took head-on the notion that the election is not till November, but yet we’re doing a poll on that now. Why are we doing that? And they said explicitly, “We expect things will change.” So they’re trying to dispel the notion that a reader should take this poll result, and take it as a projection of November. No, that’s not what we’re doing. So the reason we did this poll, is because polling data, now, can help us all understand how the public is reacting to the candidates, and how they’re thinking about the issues, what matters to them, what doesn’t, and get those kinds of themes and understanding. And I thought that that was really useful, because, I think it’s too easy to take polls, literally. And that’s really not what pollsters, or even smart reporters, covering polls are trying to do this early in the cycle.

[00:32:12]

RICK WEISS: Hmm. Great. Great example. Thanks. Gary?

[00:32:16]

GARY LANGER: Yeah, well, we too often go for the shiny new thing. And we want to grab a result that’s sexy, that makes a compelling headline that may be problematic. A lot of times, if it sounds unbelievable, it probably is unbelievable, for a pretty good reason. So we need full disclosure. We have to see a detailed statement of how the data were collected. We have to see every question that was asked, that’s being released. And we need to see the analysis, and make sure that hasn’t been cherry-picked to sell a product or point of view.

So the spadework of reporting is absolutely critical here. Good reporting can not only transmit results, we don’t want to just move information from point A to point B, like sheetrock off the back of a truck. The idea here is to add value, to add insight. And we do that through, frankly, our storytelling skills, but also our ability to add context. So we put out a study just this weekend. We did have a horse race in it. I’m guilty. But we also talked about the Joe Biden’s approval rating. And we look back at previous approval ratings. Has any president under 50 — he’s well under 50%, has any president under 50% approval, managed to get reelected? And the answer is three presidents, in the spring, before an election managed to get reelected. That was Barack Obama, George W. Bush and Harry Truman. But this is a report in which we’re able to go back to data from 1948, to inform where we’re at in this election, to show that while Biden faces serious challenges in this election, it’s been done. It’s not impossible. And while he’s weaker than Trump on the issues, he prevails on personal attributes. These are the sorts of bits of information that we can unfold, that enlighten our judgment, and tell us much more about election than we know by simply looking at the horse race. It’s been discussed, but the horse race is like the score of a basketball game in the first quarter right now. It does not predict the outcome. It may be a little interesting, but there’s much more to know.

[00:34:06]

RICK WEISS: Great. I love the concept that it’s the opportunity and the responsibility of the reporter to add value to what’s coming out here. I love that. Amelia, how about you?

[00:34:17]

AMELIA THOMSON-DEVEAUX: One thing I really appreciate about the way reporters have been writing about polls, I think, really, particularly in the past very recent couple cycles, is that they’ve been much more willing to embrace the concept of uncertainty in polling. And this is something that we thought a lot about when I was at FiveThirtyEight. And ultimately, decided to include uncertainty intervals on the polling averages to — that are presented, sort of taking, setting aside what you think of polling averages, wanting to be clear about how that there is uncertainty built into all of this. 

And I think that can be helpful because I think it’s extremely helpful because people often want polls to do things that they can’t do. They want them to be predictive, and they want them to be more precise than they really can be. And I think a huge thing that reporters can do, particularly in an election year, is really help people understand what can polls do? What can’t they do? What do we know? What don’t we know? And I think that is hugely important for building trust with news consumers, because we need to understand what are these tools that we’re using, and how can they be used, and how can they not be used? And I think putting all of that upfront, including the uncertainty, saying what we do and don’t know, ultimately, is going to make people understand if the polls are a little bit off, polls are not perfect. If the polls, right now, don’t reflect what happens in November, polls are not predictive. And I think it’s really our job, as reporters, to be doing that, and journalists, and I think folks have been doing a better job of that in the past few cycles. I’ve appreciated that work.

[00:36:21]

RICK WEISS: That’s a great point, and one we make all the time when we train reporters about covering science, because there’s uncertainty in science, too. And if you don’t admit to that in your story, then people lose trust in science, when they see results change over time, as they’re, of course, going to do as the science gets better. So it’s a great thing to understand. Okay. Let’s get into some questions here through — that we’re getting through a few channels here. And the first one is from Tasha Williams, a freelance reporter based in New York: “I’m confused a bit. Are we saying web surveys are automatically nonprobability, or are we saying participants can’t be volunteers?”

[00:37:00]

GARY LANGER: Number two.

[00:37:00]

RICK WEISS: Yeah, go ahead, Gary.

[00:37:03]

GARY LANGER: Door number two. They are certainly probability-based online panels. There’s several of them out there. In fact, there’s a little shop called The Pew Research Center that does all, or nearly all, of its domestic research using a probability-based panel. This is a panel in which participants are randomly recruited, often not even by telephone surveys, but via address-based sampling, to join the panel and take occasional surveys. The Ipsos Public Affairs has one, NORC at the University Chicago has one. There’s one at, what is it, UCLA, I think. SSRS has one. So there are probability-based panels. They’re based in probability theory, as I’ve described it to you, and they’ve been well-tested and evaluated, and they produce good, valid, reliable results. Other panels, self-selected among individuals, who sign themselves up, often under assumed names and different identities to increase their prize-winning capabilities, are not probability-based. And independent empirical testing, highly probable.

[00:38:06]

RICK WEISS: Courtney, do you want to add something?

[00:38:07]

COURTNEY KENNEDY: I would just add to that, right, so to underscore—it’s okay if the interviewing was done online. The question is where did—where were the people recruited? What was the source for the individuals that were surveyed? Where did they come from? Because the key thing is that there is no way to do scientific random sampling of the public online. Right? There is no master list of people on the Internet. It doesn’t exist. And so if you’re going to do a scientific random sample, you have to do that offline, because there is a master list of residential addresses, or landline and cell phone numbers. So you can do that scientific process of sampling. But that part needs to be done online, the recruitment. It’s okay if you ask the questions, you recruit them offline, and then ask the questions on the Internet. That’s perfectly fine. But you really got to figure out where did these people come from? And if you’re looking at press releases, describing polls, without exception, if somebody took the time and expense to use one of these probability-based online panels, they say exactly all these details: which panel I used, how that panel was recruited, was it phone numbers, was it addresses. It’s very different with the volunteer or the opt-in space. You often just see “done online”. That’s because they don’t want to get into the details of this, well, they’re actually just volunteers, and it’s a hodgepodge of convenient sampling. That’s not as pretty of a story. So you have to ask that next level question of where people came from.

[00:39:47]

AMELIA THOMSON-DEVEAUX: And you do also want to be careful to make sure that you understand terminology. Sometimes, those methodology statements from opt-in panels will say things like, “This is a nationally-representative survey.” That can really mean anything. It sounds like it’s a shiny, good thing. But it’s doesn’t mean that it actually represents the U.S. population in a, in a meaningful way. So taking the time to just learn some of the terms of our — there aren’t that many of them, so that you know what is telling you that this is a high-quality survey, and what is just a buzzword that that could mean any number of things.

[00:40:27]

RICK WEISS: Probability-based trumps nationally-representative, for example. OK.

[00:40:34]

GARY LANGER: Sort of. Sorry, Rick. Here’s the challenge. You can have a probability-based survey of a nonrandom panel, right? So I can assemble all my friends and neighbors, and then randomly select among them to take my survey. It’s just not representative of anyone but my friends and family. So if you do probability-based random selection of an opt-in internet panel, you can get results that are representative of the opt-in Internet panel. But that’s not representative what we’re really trying to measure here, which is the full population.

[00:41:11]

RICK WEISS: No wonder there is some confusion out there. There’s a lot of people sowing it, as far as I can tell.

[00:41:18]

GARY LANGER: Right.

[00:41:18]

RICK WEISS: OK. Let’s get to another question here. This is from David Scharfenberg from The Boston Globe: “Amelia, when you say you should include uncertainty intervals, do you mean margin of error or something else?”

[00:41:30]

AMELIA THOMSON-DEVEAUX: So in that case, I think I was talking specifically about the FiveThirtyEight polling averages, which are a visual, and margin of error was added to the visual. As a writer, you have to be careful to make sure that you’re giving people information in a way that they can digest. And part of that means also not hitting them with a bunch of numbers and terms that they don’t understand. So it’s not necessarily the case, that you have to include what the margin of error is all over your story. It’s more of a skill, I think, of writing about uncertainty, when it’s relevant, and not overstating your case, being aware of what the margin of error is. And, for example, if what you’re seeing is 48% of Americans think something or other. That’s close to a majority. You don’t want to say it’s the majority, though. So being careful in the words that you choose, and understanding what the margin of error means. And what it means for your analysis, as you’re writing, I think is more important, because it will affect whether you say a candidate is leading, whether you say that a majority of people think something, whether you decide to say that it’s three quarters of Americans, who believe something, that I think is the key there.

[00:43:01]

RICK WEISS: Great. I’ve got a question here that’s directed to Courtney: “How can news organizations assess whether a pollster is credible? Should they just avoid covering polls from newly-established pollsters altogether to be safe?”

[00:43:17]

COURTNEY KENNEDY: Yeah, I do think looking at the pollster’s track record is important. And I certainly have much more confidence in organizations that have polled several cycles, and have put out reasonable data. I wouldn’t have any sort of blanket rule that I would never trust a newcomer. But if you’re a newcomer, and you don’t have that track record, I think that the obligation for them to make—explain why their methodology is trustworthy, is that much higher. They would really have to explain to me that they used a rigorous sampling frame, rigorous modes of data collection. They know how to weight their data. So if all those things were in place, sure, it’s okay that they’re new. The newness itself is not the problem. I think probably one thing that that I didn’t make explicit, is that the new — the folks that come on the scene, just sort of out of nowhere, are often using really inexpensive, shoddy methodology. So the newness has, in recent years, gone largely with shoddy methodology. That’s why you hear my skepticism. But it doesn’t have to be that way. But if somebody is new, I would say that their bar of explaining and demonstrating their rigor should be quite high.

[00:44:40]

RICK WEISS: Hmm. A question as well for Gary, which just disappeared. But it what it was: “You had mentioned that state-level data can be less reliable than national data. Can you explain what happens with these subsets?”

[00:44:59]

GARY LANGER: Yeah, well, that’s specific to vote preference data. And there’s really a few reasons for it. One is a lot of the organizations putting out state-level polling don’t have the expertise, perhaps, or the budgets, perhaps, to do the level of rigor of research that’s really required for good estimates. Serious survey research is seriously necessary and seriously expensive. And surveys, often state-level surveys, that are done on the cheek, can have problematic estimates. But there’s other — there’s structural challenges as well. People move around. We get a lot of people these days if it’s — depending on the instrument, but — or the mode. But on cell phones, for example. If a survey typically will be done, largely, if not entirely, by cell phone. But people have cell phones that they got in state A, and then they move to state B. And if you’re using a sample of cell phones that have the—that they’re associated state residents with state aid, we don’t know that you moved, we can’t reach you. You’re out of the sampling frame. And there are people who’ve moved in, therefore, they’re people who’ve moved out, that are really hard to get. And there’s actually a lot of moving around. So it’s hard to get a good sample. Some address it by trying other methods, like registration-based sampling, in which you go to the state, and you get a list of every registered voter. Then you pay a company to match that list of voters and addresses to phone numbers, and hope you can reach people. But there’s a lot of bad matches, and there’s a lot of non-matches, and that produces non-coverage. Remember, I said that you want a sampling frame in which everybody’s got a chance of being selected. These registered voter lists, there’s large portions of the population that have no chance of being selected. And that can create inaccuracies. And that’s quite common in state-level polls as well.

[00:46:48]

RICK WEISS: Hmm. Anything else on subsets? OK. Got a question here, asking whether methodology of how a poll was conducted can actually skew poll results. And actually, I want to add onto that something, I, myself, had heard from someone recently about something called “sensitization”, that I think might be relevant here. Maybe that’s what you were talking about with your marijuana questions, Amelia. But can you address the question of whether methodological tweaks can actually help skew results in the direction that are desired.

[00:47:27]

AMELIA THOMSON-DEVEAUX: I mean, methodological choices can absolutely affect the way that surveys come out. And that’s one of the reasons that it’s so important to make sure that you understand how a survey was conducted, and that you have a transparent methodology statement. Because, often, when decisions are made in terms of sampling, in terms of how people were reached, in ways that that could affect the result, those details won’t be disclosed. So I think it’s just underscoring the importance of transparency. And again, also the importance of educating yourself, as a reporter, about what some of these terms mean, so that you can unpack a methodology statement, and understand what the key components are.

[00:48:21]

RICK WEISS: Great.

[00:48:21]

COURTNEY KENNEDY: I would, I would second that, and just give a few examples. So one way to do polling is what’s known as robopolling, or interactive voice response. It’s where you get a phone call, and the answer, and all of a sudden, you’re talking to sort of an automated sequence of questions. That’s still in use. It’s not super popular. But you won’t be shocked that that if you do a poll with that methodology, the people who participate tends to skew very white, very old, really Republican. Right? So that methodology choice would leave a sample that probably looks a certain way. Contrast that with if you do a poll, especially, especially like an opt in a volunteer online poll, those can skew very progressive, very democratic because who are you getting? You’re getting people who, by definition, are online, comfortable doing online tasks, excuse progressive, excuse urban, and all those kinds of things. Now, if you’ve got a really good statistician, you can try to weight your way out of that. But I would just second what Amelia said, that how you design your poll can absolutely have an influence on the types of people that get into your sample. And good pollsters can navigate that, but that’s where experience can play a huge role.

[00:49:41]

RICK WEISS: Gary, I have a question directed to you, that’s also about influence, but at the other end of the pole: “Is there evidence that exposure to news stories about polling results actually drives people’s voting decisions? And if so, how should reporters consider that, or not, when they report on polls?”

[00:50:00]

GARY LANGER: Well, let’s ask, I don’t know, President Giuliani. My point is there’s any number of candidates who had the lead in both preference polling, at some point in the campaign, who have not won their elections. People make up their own minds, and they come to their decisions over time. And they change their preferences based on the information they receive as campaigns unfold. That’s why they’re interesting and important to track. But if there was something, what I think is being described, is called “the bandwagon effect”, that once somebody has a lead, then everyone thinks, oh, these, this must be the winner, and they all pile on. If that were the case, anyone who ever led in a poll would never look back, and that just doesn’t happen.

[00:50:42]

RICK WEISS: Hmm. Okay. I do have a follow-up question here from David Scharfenberg from Boston Globe, asking Amelia if you can explain why you believe models, like the AP’s, which mix probability and opt in, and still be rigorous?

[00:50:57]

AMELIA THOMSON-DEVEAUX: So I’ll say that that’s specifically with regard to Votecast. The other surveys that we do are entirely probability-based. Votecast is different. Because it’s a, it’s a truly massive survey. It’s tens of thousands of interviews. And we combine the probability-based interviews with the opt-in interviews using a weighting methodology that allows us to get accurate results. So we wouldn’t do this with a smaller survey, a normal survey of a thousand people, that you might see published at this point in the cycle. But because of the size of the opt-in sample, combined with the probability sample, we’re able to develop a methodology that is much more accurate, and is able to skirt some of those challenges.

[00:52:01]

RICK WEISS: All right. And we got time for a couple more questions. I’m going to remind reporters, just before I get to these last few questions, when we do log off at the end, when you do leave, you’ll be confronted with a short survey, itself. Only three or four questions. Half a minute to fill it out. It really helps us. Please take that half a minute to fill out that survey about this survey briefing. And help us be better as we create briefings going forward. I do have a question here from Jocelyn Brandeis, a freelance reporter based in New York: “How do you keep the questions when you’re designing surveys? I think this is about pure enough not to skew poll results or methodology. In other words, how do you ask the question so as not to influence the outcome of the poll?” This, I guess, is the art and science of poll and survey making. Some of you want to address that?

[00:52:56]

GARY LANGER: There’s a rich literature on it, going back—I mean, the seminal work is Questions and Answers in Attitude Surveys by Schuman and Presser, probably 40 years ago now. There’s, but there’s a deep well. We stand on the shoulders of the academic researchers who’ve come before us, and done extensive empirical research, without a dog in the fight, without a profit motive behind them, to try to find out what methods work and what don’t, in questionnaire design, in data analysis and survey methodology, and in so many other areas of the complexities of the work we do. And their work is essential. And informing ourselves on it, and being well-informed is really the — a necessary hallmark of a responsible and reliable survey researcher. So we study the literature. We come to understand it. I know that many of us are members of AAPOR, and WAPOR, The World Association for Public Opinion Research. We participate professional conferences. We share our data. We’re constantly learning. It’s what’s what makes it such an interesting and fascinating business. As a consumer of surveys, or as a reporter evaluating surveys, it’s really pretty avid, if a question is biased or misleading, or has the potential of skewing results. And that’s why you have to get the questionnaire, and you have to look it over and see, is it flat? Neutral? Are they’re balanced response options? Are there reasonable coverage of the potential responses that we may want to get? Read through the questionnaire? You can tell.

[00:54:29]

RICK WEISS: Great. I want to squeeze in one last question before we wrap up. It’s very interesting and maybe useful to people in newsrooms. “Is it possible for reporters at newsrooms to conduct polls to inform stories, not necessarily on which candidate would win, but for example, on political behavior? And if so, what would it take?” And I’ll add onto that. Do any of your organization’s work with individual newsrooms to help them develop polls, so that they have a little more validity than they might otherwise? But is it reasonable for a newsroom to pull off something like this, or are they just not expert enough?

[00:55:04]

COURTNEY KENNEDY: I think you have to hire a professional data collection firm. I can’t think of any exception. But it’s — so it’s doable, but use professional help, I would turn it over to Gary or Amelia, because you guys actually I think work with media outlets more than I do.

[00:55:20]

GARY LANGER: Well, the survey research is a complex undertaking. It takes a lot of expertise and a lot of study. And no, I don’t think you can do it at your — out of your garage. But it’s possible to reach out to academic researchers, who may have good methodology and resources. There are survey-based research centers that may be interested in teaming with local news organizations. I certainly think there’s possibilities. It doesn’t always have to be just hire it, done. If you’re going to hire it done, then again, before you start, you need standards. You need to do your research, and understand what is and isn’t valid and reliable survey research. How and why? Again, I don’t presume to set those standards for others, I simply suggest we all need them. Only then, when you got your head around it, can you then go forward and see what you could do?

[00:56:08]

RICK WEISS: Mm-hmm. Anything there, Amelia?

[00:56:11]

AMELIA THOMSON-DEVEAUX: No. I mean, the way that we do it at the AP, is that we work with folks at the National Opinion Research Center at the University of Chicago. Their survey research experts, we work extremely closely with them in questionnaire development and analysis of the data. They’re really full partners. And I think we bring expertise in being able to bring the information to readers, in understanding what questions have news value, trying to figure out how to ask questions that will be illuminating for the moment that we’re in. But it is really crucial to rely on experts, to help with this work, particularly when it comes to the actual on the ground work of fielding the survey. That’s probably not something, I mean, certainly not something that we do on our own.

[00:57:10]

RICK WEISS: Great. This has been such an informative briefing. I’m going to do a last thing here that I’d like to do at the end of the briefings, and just ask each of you to take 20 seconds, and give a take-home message to reporters. If there’s one thing, maybe it’s a quotable thing, that you want reporters to walk away with at the end of this briefing, what would that one thing be? Courtney, you’re first.

[00:57:30]

COURTNEY KENNEDY: Sure. To end on a positive note, we’re fortunate in that there are good, rigorous, trustworthy pollsters in our country on both sides of the aisle. Fox News, Wall Street Journal, are a couple examples of outlets that have really good polling shops that have been doing it for long time, use good methods. And then you’ve got CNN, ABC, WaPo, New York Times. There’s a lot of good pollsters across the political spectrum. So that’s — I think that’s great, because it gives you — allows you to, to look at polling, not just through the lens of one political lens or the other, but to see areas where pollsters representing a range of media outlets, find a lot of commonality. So I would look for that. And, again, look under the hood to see where people are being recruited, and look for those more rigorous methods.

[00:58:26]

RICK WEISS: Great. Thank you. Gary?

[00:58:28]

GARY LANGER: I’ve said this before, but news organizations, for far too long, have indulged themselves in the lazy luxury of being both data-hungry and math phobic. We see a number. We want it. It supports our story. It raises us above mere anecdote. We slap it in. We move on. It’s really not acceptable. My generation of journalists has not performed well in its task to report meaningful, valid, reliable survey data, and to know the difference between that and the junk data that surrounds us. I really urge, and call on today’s reporters, to take the effort to do due diligence, to check it out before you report it, like you would anything else.

[00:59:10]

RICK WEISS: Great. And Amelia?

[00:59:12]

AMELIA THOMSON-DEVEAUX: And I mean, in addition to what Courtney and Gary said, just to repeat, context is so important. It’s really tempting to look at the results of a single poll that seem like they’re super interesting, that look like they’re going to generate a lot of interest and to just write a story off of that. But it’s really incumbent upon journalists, to contextualize findings, to try to put your critical eye to it, and see does this fit with trends we’ve seen. Are there other polls that have shown something slightly different, and why would that be? What can we find in terms of public opinion data, or other reporting, that can help explain the why of this finding that you find interesting? The polls, like, I think date people really like to have a data point that explains things. But at a moment when trust in the media is low among a lot of Americans, helping them understand, really, what public opinion data means, and what it what it can and can’t do is so, so important.

[01:00:29]

RICK WEISS: Fantastic. This has been such a great briefing, and I think really of great practical value to reporters out there. Thank you all three for doing it. Thanks, reporters, who have been here today for the work you will do to follow the advice of these experts. Check out the SciLine website for a couple of facts sheets we have that, some of which were done in collaboration with the American Statistical Association, that will also help you as you cover polls and surveys. And follow us on social media @RealSciLine. Thanks, everybody. And we’ll see you at the next SciLine media briefing.