Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
What Pollsters Worry About
I found this article really informative: Flashpoints in polling, by Pew. They cover many of the issues that have been debated in the wake of some significant recent pollster fumbles. Here are some of the highlights, with links to the research in question:
Contrary to claims that the decline of landlines has polling “teetering on the edge of disaster,” there is solid evidence that well-designed telephone polls are as accurate, if not more so, today than they were a generation ago.
How big of a problem are out-of-area cellphone numbers for pollsters? The answer depends entirely on the geographic population they are trying to target … While some technical solutions are in the works, presently there is no surefire way to capture people with out-of-state or out-of-area numbers in subnational telephone polls of adults.
To figure out which candidate will win, you need to speak only to registered voters who will actually take the time to vote. … Recent research from Pew Research Center finds that models that include information about people’s past voting behavior outperform others. Other researchers have come at the problem from a different angle. Rather than trying to perfect a likely voter model, they focus instead on fielding large samples and weighting them to very detailed models of the electorate. Specifically, some researchers have hadsuccess using advanced statistical techniques, which have catchy acronyms like MrP (multilevel regression and post-stratification) and BART (Bayesian additive regression trees). … Poll consumers interested in the cutting edge of opinion polling would do well to follow firms testing out these types of approaches.
A recent study by our methods team looked at nine online surveys conducted by eight different vendors and found their results varied widely. The results from the top performer looked relatively good when compared with high quality government-sponsored benchmark surveys. Others performed less well. One key finding was that extreme caution should be exercised when using online polls to look at data among key racial and ethnic groups, as estimates for blacks and Hispanics based on the online sources tested were particularly inaccurate.
AAPOR recently produced a comprehensive report on how survey researchers are trying to make use of big data. Pew Research Center launched Data Labs in the fall of 2015 to begin experimenting with new sources of data and new analysis techniques including machine learning, natural language processing and web scraping in order to supplement our more traditional lines of work in survey and demographic research.
Can we trust the polls? Their answer is, “Maybe.”
Does any of this suggest to you any significant way the general election results might deviate from the polls?
Published in General
Using cell phone numbers can complicate accurate data collection, as many people have numbers which don’t correlate to where they actually live. For example, I have a Texas cell number but these days I reside in the Pacific Northwest.
Yes, they discuss that — here are some of the ways they’re trying to compensate for it, but I think it’s still a big problem. The Pacific Northwest? How are you coping? I figure a pollster could tell pretty easily if you were really from there: “Sir, which of the following items is not recyclable: a SunChips bag, the new Odwalla plastic PlantBottles, or an old Dasani container?” If you hesitate, you’re from out of town.
Then there is this. Some polls appear to have a dose of MSU. (I don’t mean Michigan State University, either.)
Seawriter
A public service reminder: For the sake of our country’s future, be sure to lie to pollsters.
I’m currently living in the Spokane suburbs and am coping well, thus far. Politically I don’t feel too out of place, as Eastern Washington is almost as conservative as West Texas.
As for your question, I’m guessing the SunChips bag is not recyclable.
Funny story: last week I drive down to St. Maries, Idaho, where my grandmother was born back in 1912 and where my great-grandparents are buried. After I had spent a few hours in town, I started to head back. But on the way out, I stopped to visit a monument dedicated to John Mullan, a U.S. Army officer who built the first overland road linking western Montana, northern Idaho, and eastern Washington back in the 1860s. About a minute after I parked and as I was getting out of my truck, a local (seeing the Texas plates on my pickup) pulled up and asked me: “Are you from Texas?” “Yes,” I replied. He then asked, “Are you voting for Trump or Hillary?” “Trump,” I answered. “Great!” he exclaimed. “You can stay.” Then he drove away.
Oh, so you get to see the sun occasionally! Yeah, that’s practically another state.
I wonder how services like Nomorobo are going to affect telephone polling. As I understand it, Nomorobo blocks automated calls that are made to multiple phones from the same phone number. With the service, I almost never get political calls, charitable solicitation calls, or polling calls on my landline. They’re currently testing a similar (pay) service for cell phones.
Are you gonna be up there in early-mid November?
This is an interesting blog post
May it be true.
It was an interesting article. Its a more difficult issue than I really appreciated.
I’ve spoken to a few people who do polling. As I understand it, there is often a little bit of judgment and “experience” thrown into these models, so its often polls + a factor. But I just wonder about the polling industry. Once we are out of the primary season and there are no immediate votes to predict, I wonder if some of these polling companies have an incentive to may show a counter intuitive finding– showing Trump in the lead or showing Hillary losing support with an unlikely population would be great for clicks, and there is no way to gauge its accuracy. So I wonder if it happens. I’m not saying its systematic, but I wonder if this is a thing.
The pollsters I know make their reputations (and thus their livings) by getting it right — you’re only as good as your predictive abilities. So I think few would skew their results on purpose. It makes them look incompetent or not-worth-the-bucks next time if they get it wrong.
No fortune-teller ever went broke telling suckers what they want to hear. Similarly, no polling company ever goes broke telling their client what they want to hear. Pollsters are hired for the same reason as fortune-tellers – to give the prognostication desired by whomever is paying the bill.
It is no accident that with the press (one of the biggest customers for polls) skewing so far left that so many polls skew left. Like a good fortune-teller a good polling company can always find reasons to weight the factors clients want to hear higher than the factors clients do not want to hear.
By the way, this goes on in science, too. Studies show what those paying for the studies want to hear. That is how to get your grant renewed.
Seawriter
Yes, I will.
There’ll be a meet-up. Sounds like anything you like? It would be post-election, which would involve more drinking than usual, either way…
I would bet that Spokane gets more sun than most of East TX (West TX is a toss-up).
Well, is there any watchdog organization measuring the results of at least the most famous polling firms to see who’s accurate, who’s not, &c.?
Does anyone know of any talk at least among conservatives to organize something like that?
How about the kind of polling done for private companies? Do you think competent predictions–which would include, it seems to me, some honest-to-goodness talk about the limits of predictability–are to be found there more than in political polling?
Why should there be? (I know, because the government regulates everything. But polling is protected by the first amendment – at least for now – and is telling the sweet lies the news media wants to hear.)
It depends upon what the private company wants to hear. If they really want an honest assessment, that is what they get. If they want their own instincts confirmed, that is what they get. There have been some polling disasters when that happens. (Think Edsel or J. C. Penny’s lurch left some years back.) Private companies have a motivation to seek accurate polling. A bad business decision (based on a skewed poll) has potential to put them out of business. Major media corporations and governments lack that grounding. They can screw up forever and keep their doors open.
Seawriter
I don’t mean the government should be doing it–as per my other suggestion. It should be done by conservative organizations because it matters to conservatives!
Precisely because major media corporations are called the enemy by a significant part of the anti-liberal Progress chattering classes, it might be useful to do the work. But by your own argument, maybe conservatives of any kind of organization don’t want to know the truth?
Yes, but who will bell the cat? What you are talking about requires time and money. You can substitute some time for money, but even crowdsourcing this type of effort would require the concerted full-time efforts of several people. The problem conservatives face in spending all of their free time on something is – unlike many SJWs – they have actual lives and more remunerative things to spend their free time on. (Remunerative in more than the monetary sense.) Find someone to underwrite it and it would possibly fly, but most of the potential funding sources tend to be invested in getting lied to by
fortune-tellerspollsters.Seawriter
Sir, I say this with respect: For tactical reasons–though not only for tactical reasons–I’m here on Ricochet to think about what future there may be to America. So I discount explanations for why things aren’t happening in favor of explanations for how things might be done. So also, I’m more skeptical of explanations for decline than I am of explanations for improvements.
I’m not sure if you think this is prudent or commendable; or if you share the attitude. But that’s what I’m here for & that’s why I’m saying, this can get done. Americans have done far more with far less. If some kids can invent PayPal, this can be done, too.
I understand why someone running a campaign is interested in poll numbers. I don’t understand why anyone else is.
Even if 100% accurate it doesn’t mean anything to me.
Claire,
What I am hearing about this is that it gets more expensive and time-consuming to do a poll with a large enough sample for accuracy. Many pollsters appear to be publishing quicky polls with only a few hundred sample size. I think it needs to be over a 1,000 to get reasonable accuracy. I think the cell phone thing is part of this too. If someone calls you on your cell phone from a number that you don’t recognize often you’ll just let it go to voice mail. That means they’ve got to make x more calls to get their sample size up. If they are trying for a quicky overnight poll they may fudge with a sample size too small.
Perhaps this is an urban myth but it makes great sense. Cell phone behavior is different from land line behavior.
Regards,
Jim
A colleague who was a former TV journalist and current Associate Professor of Communications tried to convince me of two things:
We didn’t argue too much about it because it was just too stupid to consider. I told him he was wrong on points of fundamental rights, privacy, and scientific foundations, and then we had a beer. (Lots of my close friends are Democrats.)
This all comes back to me w/ @claire’s post. The longer I work in higher education and do social science research, the more I find that all research is subjective, no matter how careful one tries to adhere to the assumptions and protocols of scientific methodology. More often than not, essential assumptions are violated which means that results are not founded in science. We are right to question polls, their pools, their wording, and their statistical analysis.
But why bother? Why should I even look at a poll ever? I’d rather read the horoscopes.