The Problem with Polls
We go through this every election cycle, but it's worth repeating: many polls are unreliable. But it's impossible to know which ones.
Most news organizations lead their political coverage with poll results, as if that's news, and breathlessly try to spin useful information out of a blizzard of numbers. Very few of them bother to figure out exactly how flimsy some of the polling methods are.
There's a new ABC/Washington Post poll out. It "reports" that Gingrich and Romney are each at 30%. It "reports" that President Obama's approval numbers are up.
It surveyed about 1000 people. Roger Simon, in Politico, expands:
Most people do not read the fine print. Just as most people, including most journalists, have no idea what the “margin of sampling error” really means, though most good polls tell us that. (The Post-ABC margin was plus or minus 3.5 percentage points for all adults.)
As I said, this poll has a very good reputation and I “believe” the results in that I believe they were calculated carefully and (unlike some partisan or campaign polls) without any agenda.
But in the vast, murky world known as reality, are Gingrich and Mitt Romney really tied 30-30? And, if they are, what different does it make? The primary is not a national contest, but a series of state contests by which the winning candidate amasses a majority of the approximately 2,288 pledged and unpledged delegates to the Republican National Convention.
Does Obama really lead Gingrich by 8 percentage points in a (currently) imaginary matchup?
I dunno. Sounds right to me. But I am an even smaller sample than 0.0003 percent.
Gingrich’s drop in the polls — one shows him in third place in Iowa behind Ron Paul and Romney — and Obama’s rise have become big media talking points over the past few days.
There is little real political analysis anymore. Instead, there are journalists who read polls and try to explain the results: Newt’s drop? Attack ads by his opponents are damaging him, people are learning more about him and don’t like what they are learning.
Obama’s rise? The 21st paragraph of a sidebar to the Post-ABC poll contains a figure that may be of critical importance: “The new survey finds that most Americans are optimistic about their personal finances, even though gloom continues about prospects for the national economy.”
People who are personally optimistic are the kind of people who do not change horsesmidstream, especially if they feel the stream is strewn with rocks.
Is this how “most Americans” — based on a survey of 1,005 of them — really feel?
A lot of political arguments, these days, tend to center around poll numbers. You hear it on Fox News and CNN and the Sunday talk shows -- even around the dinner table -- people confidently asserting what the American people think, feel, will or won't stand for.
It's witchcraft. It's voodoo. And it's a terrible way to run for president, make an argument, or settle a dispute.
But it's all we've got.
- Comment (9)
- · Quote
- · UnfollowFollow (2)
Comments:
May '10
Re: The Problem with Polls
The key, I think, is to stick with one reliable polling firm. I've got Rasmussen bookmarked and pretty much ignore all others. That way even if the overall data is skewed in some way due to polling philosophy, you'll still get a pretty accurate sense of trends.
Dec '10
Re: The Problem with Polls
I like RCP's average. They're method seems to dampen the craziness a little.
Jan '11
Re: The Problem with Polls
Well, polls may measure how the group is responding, but they offer no argument for or against a candidate, or for or against a proposition. They're intellectually vacant.
Measuring the electorate's collective response is what you do after the question has been answered by each individual. It shouldn't matter to each individual while they're making up their minds.
Of course, polls (like the Force) have a powerful effect on weak minds.
In a healthy electorate, each individual should decide for their own self-interest. It's dysfunctional for individuals to pretend they know what others are thinking, or they know what others need, or they know what would be best for others. That's for others to know and decide for themselves. Instead, be selfish. You're the one who knows best about what you need and want, so stick to what you know.
Political analysis based on polls is just the reporter's way of avoiding the real job; namely, to present the candidates' arguments and reasoning. And see the perverse effect. Since reporters report on polls, candidates stop "reasoning" and instead offer what will get them notice - poll-tested wah-wah.
Apr '11
Re: The Problem with Polls
Of course Obama's numbers are up. His mouth has been shut for...what...5 minutes now. The longer he keeps it shut the better the numbers. His problem is the proverbial "fool" problem. "It is better to be thought a fool than to open one's mouth and remove all doubt". Obama historically has felt the need to remove all doubt.
As to polls in general, I consider them all circumspect unless I voted in them. As a rule the only poll I care about is the one in November.
Mar '11
Re: The Problem with Polls
I hope some of our poli-sci professors (Prof. Groseclose, especially) will respond with some specifics on methodology, but here's my thumbnail response.
Rob's right. Polls, even when well constructed, are flawed. It is difficult enough to construct poll questions to properly measure what you intend to measure, and its often extremely difficult to interpret the results. Its not only that the polls are "voodoo", it is also that people often answer two questions in completely contradictory ways. In part this is why we end up with "Right Direction/Wrong Direction" questions carrying so much weight. Most (good) pollsters know this and admit it frankly. Within campaigns, usually, these things are taken with a grain of salt (but they aren't ignored).
But what is dangerous, frankly--and, Jim Ceaser's book on the 2000 election documents this--is the way the 'amateur' political prognosticators, especially in the mainstream media on TV, use poll numbers exploitively to distort the perceptions of their viewers. Polls serve the horserace narrative that sells TV news, and so the networks don't include the proper caveats when they present this information.
Apr '11
Re: The Problem with Polls
I think the value of polls is to show trends. Most people overreact to news initially, so you need to track their opinions for a while to come to a meaningful (if any) conclusion. The Health Care Bill is a good example. It continues to poll badly, showing how awful it truly was. People tend to forget about bad legislation since the real costs get buried, but not in this case as the ramifications are real.
I'm not sure what to make of approval ratings. First of all, I can't believe 50% of Americans approve of our president, but I guess he is so much of a Cult of Personality that some people are just charmed by him. The real question is, how many of those people will go out and vote? I think conservatives/libertarians/independents are ready to come out in large numbers to get rid of him, but that could simply be wishful thinking. Beyond that, let's look at his numbers in swing states. They aren't looking too good. He can shore up California and New York all he wants, but he'll need to carry states like Ohio and Florida to win.
Aug '10
Re: The Problem with Polls
OK, consider yourself answered.
In terms of question wording, identifying likely voters, people having cell phones, people being embarrassed to give the "wrong" answer, opinions changing closer to the election, etc, those are all valid concerns.
The issue about sample size as a proportion of population is a complete and total canard. The only thing that matters for sampling error is sample size. Period. Population size doesn't matter at all. You need the same sample size to get a good measure of Iowa as you do of the United States as a whole.
Aug '10
Re: The Problem with Polls
To expand on my previous comment, let's put it this way. Sampling error is how far off you are by virtue of only interviewing a relatively small number of people. This is what the "+/- 3%" thing means. It's a luck of the draw thing. Sampling error decreases proportionally to the square root of sample size. Neither population size nor ratio of sample to population has anything to do with it.
On the other hand, if there's something systematic about your procedure then this is both harder to quantify and won't be solved by interviewing more people. For instance, if you have a push poll with really biased wording (eg, "Don't you support efforts to save polar bears from the greedy oil companies?") this bias won't be solved by asking the question to a million people instead of a thousand people. All that'll happen is you'll get an asymptotically more reliable result of how people respond to a biased question.
Edited on December 22, 2011 at 9:12pmMay '10
Re: The Problem with Polls
I think what most people don't understand (or believe) is that you can get pretty good accuracy with a relatively small sample---and increasing the size of the sample doesn't add that much accuracy, unless you increase the sample by huge (impractical) amounts. Statistics is largely counterintuitive and that is why, I hypothesize, it really wasn't developed until the 20th century, unlike regular math, which was mostly worked out in ancient Greece.