Let’s face it, this is the question on everyone's mind. Jay Cost and the folks at HotAir and National Review have been hashing through the state of election polling, and find it falls short of their expectations. Pat Sajak had his own take here at Ricochet.
Polls are criticized from the Left and the Right, and even the middle. Bias is claimed from all corners.
So, are all polls rubbish? No . . . but they are all estimates of a snapshot in time. Often very good ones. But any poll is an educated guess, not reality. (Let’s leave aside in-person and mail surveys right now.)
Sure, traditional live-caller telephone polls begin with a very rigorous foundation in probability theory, a lot of math that will tell you your probability of being right with very great precision. But notice, even here, in the perfection of the abstract, you will get the incorrect snapshot of opinion sometimes. It’s just a very good estimate. And any given result might be wrong. This isn’t so important if we’re looking at, say, a 70/30 vote split. But when we’re talking a handful of points, small variations make all the difference.
Then you add in the messy imperfection of the real world . . . some kinds of people are much more likely to pick up the phone and answer your questions, and those response rates will change depending on the day of the week.
You can’t call a person’s Social Security number; instead you call landlines used by multiple people in a household, and you survey respondents who have multiple telephone numbers. Some people (a lot now days) only have cell phones, which are much more expensive to capture, have a higher non-response rate, and have major sampling theory problems of their own. Straight Robo-calls don’t capture cell phones at all and can’t attempt to correct for all the non-response bias like a live-call system can. Robo-polls actually have a good track record, but that might be in part because they are weighting their data to the results of live polls.
So, to adjust for these known deviations from the abstract ideal of a probability sample, pollsters can ask for particular kinds of respondents in a household, call back numerous times and extend the field time (academic surveys can be out more than a month), they can pay cell phone users for their minutes and oversample certain demographics. And don’t forget, none of this speaks to unknown unknown biases.
Finally, even if you have the perfect survey of all adults, you still need to determine which respondents in that sample are going to turn out. That means a pollster asks questions and scores responses for inclusion as a “likely” voter. Unfortunately, many who look like likely voters don’t vote, and many who don’t look like likely voters do vote. In other words, more guesswork.
All pollsters must then weight their data to make it realistic and account for all the known deviations from the ideal.
Polling in practice is far from a perfect probability sample, and getting more so each year. That’s one reason YouGov, for instance, has thrown out the traditional survey sampling methodology in favor of a rigorous matching procedure that performs exceedingly well.
Nonetheless, even without a “proper” probability sample -- and with all of the non-response and other problems -- polls are pretty accurate. But this still means an average error of a bit less than 3 points. In other words, the error is somewhere around the likely margin of victory in many elections.
Polls are still very valuable, and they are the best option we have for measuring public opinion. But we need to take them for what they are and look for trends within a particular pollster’s work and standings relative to the demographic/political makeup of particular polls.
Everyone should take a deep breath, relax, and realize that no poll or average of polls is reality. It's not magic either . . . it’s just a better or worse approximation of reality that tends to get better as we get closer to the election.