How to read polls like an expert — or, at least, not like a newbie
Having already suffered 13 strokes after seeing headlines misrepresent what a poll says about a race, I decided to get to the point and ask people who actually watch or take polls to make a living deserve to offer basic guidelines for their consumption. I asked everyone to give three tips for laypeople who come across a survey; Since they were good with numbers, each came up with exactly three.
Their answers are listed below, with slight changes and underlining for emphasis. Sometimes you will find that the advice is similar. Think of their advice itself as a poll; these recurring responses are then at the forefront of importance. Even beyond the margin of error.
Survey and Analysis Editor at CNN
1. Surveys are not precision instrumentsand if you expect them to give you accurate answers, they will be of much less use to you than treating them as a tool for measuring broad public sentiment.
2. Sometimes When the polls disagree, the nature of that disagreement can tell a lot. For example, that there is a lot of uncertainty about which voters will go to a particular election, or that most Americans are not very familiar with a topic they are being asked about.
3. Transparency matters: At a minimum, every survey should include a clear explanation of who conducted and paid for the survey, how the pollsters selected and contacted the people they interviewed, what exactly those people were polled, and the steps the pollster took to ensure that their survey was conducted reflected a broader population.
Data journalist at The Economist and author of “Strength in Numbers: How Surveys Work and Why We Need Them”
1. Take the margin of error and double it. Remember that a survey is a sample of a larger population. Every pollster reports (or should report—if they don’t, it’s a red flag) a number called their “margin of error,” which tells you how wrong their poll might be based on the likelihood of it coming up with an unrepresentative sample of those polls larger group have spoken group.
But one thing I write about in my book is that a single election poll is subject to many more errors than just this “sampling error” alone. For example, there is the possibility that members of one party are less likely to vote in their elections than others (which was the case in 2020 and 2016), and there is an error in predicting who will actually vote. So, historically, the distribution of error in election polls is about twice as large as the error rate implies.
Which brings me to the second point:
2. Aggregated surveys together. Because individual polls are subject to so many errors, averaging gives you a better idea of the shape of opinion on a given issue—whether it’s about who people will vote for or whether they favor a particular policy. This also helps you avoid overreacting to inter-poll phantom oscillations that occur due to the high level of noise in a poll.
3. Finally Think of the process that created all of the surveys you look at. By closely examining an individual pollster’s methods, you can determine whether their numbers are more trustworthy. When a pollster surveys only people with landlines and doesn’t adjust (or “weights”) their sample to be representative of cell phone owners, for example, they overestimate support for the kinds of things that people with landlines favor.
But it also helps us to remember that the average pollster is generally subject to the same types of bias as all other pollsters. If the “data generation process” (as statisticians like to call it) for one poll is biased because Democrats or Republicans or the elderly, etc. don’t respond to their interviewers, then other polls are likely to fail as well. This helps you calibrate expectations for errors ahead of an election.
Survey Analyst at The Washington Post
1. Polls are not forecasts! They are snapshots from when the survey was conducted and reflect what people were thinking when the survey was conducted and not in the future. Probable voter models are just that—imperfect attempts to depict a future but uncertain population.
2. Polls can provide roughly accurate estimates of candidate support at best, but You are unable to tell which candidate will lead in close races. If the percentage of support for the two candidates on vote choice questions is closer than twice this margin of error, the difference is unlikely to be significantly different. The margin of error also does not take into account other sources of error, such as e.g. B. Different frequency of answers from the supporters of a candidate.
3. There are many interesting things to ask beyond the overall horse racing result. See how support is split by different electoral groups, and read the rest of the topline or the poll report to see what people think about election-related issues. There’s a lot to learn about what people think about important issues.
Lead Election Analyst at Thirty-five
First, remember that while polls (at least scientific polls from reputable companies) are still the best tool we have for predicting elections or measuring public opinion, They’re not meant to be super-precise instruments. Even the best polls have margins of error that are an inevitable byproduct of not speaking to every single person in the country. So, for example, if you see a candidate “leading” a poll by just 1 or 2 points, it’s best to think of that race roughly as a toss-up, since a 1 or 2 point error in the poll doesn’t mean anything not unusual.
But secondly Don’t try to beat the polls. The 2016 and 2020 elections saw publicized voting errors that benefited Republicans, leading some to conclude that polls are generally biased against them. But the voting error in the 2012 election benefited the Democrats, and if you look at the voting errors of the past several decades, you can see that they’ve bounced around unpredictably. So there may be a new Trump-era reason why polls underestimate Republicans, but it could also just be an outlier in history, so be prepared for a voting error either way (or no voting error at all, either). always). possible).
Third, Don’t pay too much attention to a single poll, especially when it’s an outlier, aka a poll that’s far from the consensus of all other polls. Instead, average them together for the most realistic reading on a race or problem.
The alleged author of this article
I’m only going to intervene here to also suggest that you pay attention why You see a poll. Is it from a campaign? Is it on a partisan site? Poll results can be confusing or contradictory, making it easy to push numbers for a specific purpose. Consider the context.
Finally, if the polling company in question spends an inordinate amount of time represent conspiracy theories on Twitter throw a few grains of salt over their assessments.