Are ALL polls rubbish?
Let’s face it, this is the question on everyone's mind. Jay Cost and the folks at HotAir and National Review have been hashing through the state of election polling, and find it falls short of their expectations. Pat Sajak had his own take here at Ricochet.
Polls are criticized from the Left and the Right, and even the middle. Bias is claimed from all corners.
So, are all polls rubbish? No . . . but they are all estimates of a snapshot in time. Often very good ones. But any poll is an educated guess, not reality. (Let’s leave aside in-person and mail surveys right now.)
Sure, traditional live-caller telephone polls begin with a very rigorous foundation in probability theory, a lot of math that will tell you your probability of being right with very great precision. But notice, even here, in the perfection of the abstract, you will get the incorrect snapshot of opinion sometimes. It’s just a very good estimate. And any given result might be wrong. This isn’t so important if we’re looking at, say, a 70/30 vote split. But when we’re talking a handful of points, small variations make all the difference.
Then you add in the messy imperfection of the real world . . . some kinds of people are much more likely to pick up the phone and answer your questions, and those response rates will change depending on the day of the week.
You can’t call a person’s Social Security number; instead you call landlines used by multiple people in a household, and you survey respondents who have multiple telephone numbers. Some people (a lot now days) only have cell phones, which are much more expensive to capture, have a higher non-response rate, and have major sampling theory problems of their own. Straight Robo-calls don’t capture cell phones at all and can’t attempt to correct for all the non-response bias like a live-call system can. Robo-polls actually have a good track record, but that might be in part because they are weighting their data to the results of live polls.
So, to adjust for these known deviations from the abstract ideal of a probability sample, pollsters can ask for particular kinds of respondents in a household, call back numerous times and extend the field time (academic surveys can be out more than a month), they can pay cell phone users for their minutes and oversample certain demographics. And don’t forget, none of this speaks to unknown unknown biases.
Finally, even if you have the perfect survey of all adults, you still need to determine which respondents in that sample are going to turn out. That means a pollster asks questions and scores responses for inclusion as a “likely” voter. Unfortunately, many who look like likely voters don’t vote, and many who don’t look like likely voters do vote. In other words, more guesswork.
All pollsters must then weight their data to make it realistic and account for all the known deviations from the ideal.
Polling in practice is far from a perfect probability sample, and getting more so each year. That’s one reason YouGov, for instance, has thrown out the traditional survey sampling methodology in favor of a rigorous matching procedure that performs exceedingly well.
Nonetheless, even without a “proper” probability sample -- and with all of the non-response and other problems -- polls are pretty accurate. But this still means an average error of a bit less than 3 points. In other words, the error is somewhere around the likely margin of victory in many elections.
Polls are still very valuable, and they are the best option we have for measuring public opinion. But we need to take them for what they are and look for trends within a particular pollster’s work and standings relative to the demographic/political makeup of particular polls.
Everyone should take a deep breath, relax, and realize that no poll or average of polls is reality. It's not magic either . . . it’s just a better or worse approximation of reality that tends to get better as we get closer to the election.
- Comment (11)
- · Quote
- · UnfollowFollow (2)
Comments:
Apr '11
Re: Are ALL polls rubbish?
Just a question..how does Rasmussen handle cellphones? What sort of adjustment do they make due to the fact that they only poll landlines? Or is there any way to know what sorts of assumptions they make?
Edited on September 28, 2012 at 10:21pmJul '10
Re: Are ALL polls rubbish?
Taking all you say in, it would seem the logical route is to simply disregard all the polls. I would personally much prefer more discussion of the actual issues than a discussion what this or that inaccurate poll says or doesn't say. I have personally gotten to the point that I simply don't pay any attention to them anymore. Speak to the issues:
- Horrendous economic disaster.
- Massive foreign policy disaster.
- Open disdain for allies. This is not only Israel, but Iraq and Afghanistan.
- Complete disregard for any opposing ideas or thoughts, with demonization of that opposition.
Keeping speaking to those things, and the polls will take care of themselves.
Re: Are ALL polls rubbish?
wmartin . . . Rasmussen apparently uses some internet respondents to fill the cell-phone gap, but he isn't very clear about what that actually means in practice.
This pushes him (and maybe other robo-pollsters?) even further away from traditional probability sampling and closer to what we do (although our focus is on movement, not a precise point estimate).
Feb '11
Re: Are ALL polls rubbish?
The other thing to always remember with polls is that they cost money. The more accurate you want the poll, the more money it's going to cost. You think news organizations have lots of money lying around these days? The wolves are at their doors. And you have outfits like PPP which use polling in order to drum up other types of business for their company, e.g., consulting which is much more lucrative. Dean Debnam, head of PPP, is a leftie political entrepreneur.
Re: Are ALL polls rubbish?
Devereaux . . . I wouldn't ignore the polls. I just think we all need to take them with a large grain of salt, and look more at trends within series of polls from the same firm, at the breakouts of the samples, and at recent elections. Polls aren't magic, they aren't the Truth, but they aren't rubbish either. Like any human endeavor or science, it is imperfect.
May '10
Re: Are ALL polls rubbish?
The web cartoon xkcd has a powerful explanation of the "margin of error for a 95% confidence level" (the conventional practice for public opinion polls).
In layman's terms, it means if you performed the same poll 20 times, you'd get slightly different answers because of random sampling differences. It's likely that 19 of them (95% of them) reflect reality within the margin of error, and the 20th is wrong by more than the margin error. With the number of polls that are performed, there are bound to be tons of them that fall into the second category.
And guess which polls generate headlines.
Edited on September 28, 2012 at 11:24pmNov '10
Re: Are ALL polls rubbish?
I am ignoring the election, in hopes that it will go away. I get too worked up thinking about another four years of Barack Obama in the White House.
But I've also got a Governor's race to worry about (in which the Republican is painted as a radical because he gave money to GWB's campaign).
And a Congressional race to worry about.
And a grown kid living at home going to college, which is really worse than all that other stuff....
Jul '12
Re: Are ALL polls rubbish?
The question is: what earthly good are opinion surveys to me, Mr. John Q. Public? I suspect that my comrades in the hoi polloi have the good sense not to pay them the slightest attention when coming to a decision, much less give them any credence. Hope that's the case, anyway.
(Has anyone ever thought they've actually observed a bandwagon effect from a published poll? Just a question. A non-rhetorical one. )
But really, figuring out if the season at Del Mar is over yet seems like a better use of time for the nonprofessional. One day William Goldman's "nobody knows anything," will be seen to apply to nearly everything and we'll all get comfortable with it.
Jul '12
Re: Are ALL polls rubbish?
Dang it. The season IS over at Del Mar. Now all I've got is this stupid election to bet on. How's Romney on a wet track?
Jun '10
Re: Are ALL polls rubbish?
What i do not get about these polls is why they ask "Do you consider yourself a Republican or a Democrat"? Why not simply ask -- "Are you a registered Republican or registered Democrat"? Why must we get into these hazy What do you feel your are today" questions. It is simply Republican, democrat or other.
Oct '10
Re: Are ALL polls rubbish?
What I want to know is, how do polls affect the morale of a given candidate's supporters?
Like the Laffer curve, there must be an 'enthusiasm' curve, a point at which supporters of a candidate are the most energized. Polls indicating an election is tied, theoretically, should provide maximum enthusiasm, because each vote is theoretically more important.