Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
Polling Bias
As I was scouting the returns on Tuesday evening — in preparation for and during the Ricochet podcast — I noticed that in a number of races the polls were way off. No poll that I know of predicted anything like the landslide Tom Cotton achieved in Arkansas; none suggested that the Senatorial race in Virginia would be a cliff-hanger; none gave Scott Walker anything like the margin of victory that he received in Wisconsin. I could go on.
There were, however, so many races taking place that I was unable at the time to discern whether this was a general trend, and by the time that I had a few moments free to look into the question (which is to say, this morning), Nate Silver had run all the numbers. Here is a taste of what he has to say:
For much of this election cycle, Democrats complained the polls were biased against them. They said the polls were failing to represent enough minority voters and applying overly restrictive likely-voter screens. They claimed early-voting data was proving the polls wrong. They cited the fact that polls were biased against Democrats in 2012.
The Democrats’ complaints may have been more sophisticated-seeming than the “skewed polls” arguments made by Republicans in 2012. But in the end, they were just as wrong. The polls did have a strong bias this year — but it was toward Democrats and not against them.
Based on results as reported through early Wednesday morning — I’ll detail our method for calculating this in a moment — the average Senate poll conducted in the final three weeks of this year’s campaign overestimated the Democrat’s performance by 4 percentage points. The average gubernatorial poll was nearly as bad, overestimating the Democrat’s performance by 3.4 points.
You should read the whole thing — because part of his point is that 2014 was the mirror image of 2012. Last time, the polls skewed towards the Republicans; this time they skewed towards the Democrats.
Polling is, as a colleague of mine remarked a day or two ago, a competitive business. Those who succeed in predicting with considerable accuracy the results in an election have an edge in getting work from private-sector operations interested in sampling opinion. Those that go awry lose business.
But it is also exceedingly hard to get things right. In the weeks leading up to this election, I frequently went to Real Clear Politics to see what the polls were predicting. In my gut, I thought that there ought to be a wave. Obama’s failures were legion. Thanks to Harry Reid’s handling of the Senate, there was no daylight between the incumbent senators and the president, and Obama kept doing the Republicans’ work for them by insisting that the election was a referendum on his policies.
But my gut instinct in 2012 was similar, as some of you will remember. At that time, I thought that the polls were skewed and that the handwriting was on the wall for Barack Obama. Given his performance, I said to myself, how can it be otherwise? And Romney had trounced him in the first debate in a way that I had never seen a presidential candidate trounced.
As it turned out, I underestimated the effectiveness of the Democrats’ turn-out operation. I underestimated the folly of young voters. And I underestimated the ineptitude of the Romney campaign.
But I had been right two years earlier, in 2010. In fact, I predicted a Republican landslide for that year in September, 2009. This had nothing to do with the polls at the time and everything to do with my judgment of the significance of the Tea Party insurgency. Obama intended a revolution, and the opposition had formed.
In 2012, I trusted my gut and I was wrong. This year, I held back from making predictions on the conviction that 2012 had demonstrated that I had not kept up, that the country had changed, that sluts really did vote, and that my understanding of the American character was overly optimistic. And so, this time, I was overly pessimistic. The generic voting polls suggested that the Republicans would win in a landslide; the polls for particular races suggested that the landslide might not affect the Senate and the gubernatorial races. I doubted whether I could make an accurate prediction — and this time I was wrong in not trusting my gut.
So what is the takeaway? First, the polls are often wrong. It is really hard to predict what will happen because it is really hard to predict who will turn out. This is true in midterm elections, and it is no less true in presidential elections. Second, my gut is not terribly reliable either.
We live in a time in which things are fluid. This year, the Republicans got a majority of the Asian vote. That is a real shift, and I know of no one who saw it coming. This year, the Republicans received 36% of the Hispanic vote. That is telling as well. This year, when Republicans had many more gubernatorial chairs to defend than the Democrats, they added dramatically to their strength in that regard. And with regard to the state houses and the state senates they did better than at any time since the 1920s.
Elections are a bit like the stock market. Past performance does not predict future returns.
Published in General
Member Snirtler had some good insights in the comments of a similar question I asked on the Member Feed.
I’m particularly interested in how pollsters got Georgia so terribly wrong. I still don’t get it.
The takeaway for me is this: human nature has not changed. Government is still the least bad option when a Hobbesian war of all against all is the only other choice, but just barely so. Whichever party is currently demonstrating the inefficiency and inefficacy of government will lose; back and forth the pendulum will swing; the merry-go-round will, forever, go round and round.
We may perhaps see a longer change to Republican control due to the extra energy imparted to the pendulum by Obama’s truly terrible policies. His doing so was only possible thanks to the hefty push given it when Bush and Republicans did the same thing between 2000 and 2006.
I suspect that the natural tendency to overcompensate for past mistakes is a contributing cause here. Pollsters overestimated R in 2012 so they adjusted and ended up overestimating D in 2014. Happens all the time. For instance the US intelligence community throughout the Cold War went thru wild swings of overestimating and then underestimating Soviet capabilities. Same thing happened with WMD in Iraq in Gulf War I and II. It’s hard to get it just right
Marquette University Law School sponsors a poll that has been remarkably accurate since it began back before the 2012 recall elections. They correctly predicted Walker +7 in the recall. They had Wisconsin as +8 for Obama in 2012 (actual was Obama +7). And they actually over estimated the margin of Walker victory in 2014 at +7. Every other poll, however, had the race a dead heat.
Weren’t the polls in 2012 correct? At least Nate Silver’s were. I don’t think his skewed Repubican, nor did most, which is why many of us thought they were skewed for Dems.
Silver had MD going Democratic by 9.4 points. Hogan won by 7+. A 16 point swing is ridiculous.
My colleagues tell me that 2012 was an anomalous year and I think the midterms bear that out. I’m not aware of anyone doing a definitive post-mortem on it but I believe the consensus is that the Dems had an usually effective knock-and-drag operation. We started noticing here in California early in the year that this was not reproducing itself in several special elections.
That being said, it’s more important that you look at the sample and how they contacted respondents than just looking at the topline numbers. The numbers are a reflection of the sample, which is a reflection of the assumptions about who will actually vote. Polls that rely on Random Digit Dialing tend to skew more Democrat. You can get a better picture with data from a voter file but you still have to set quotas based on how you think the electorate is divided along partisan and demographic lines.
I also think it’s difficult to tell what’s going on because we usually don’t have access to internal campaign polls which I believe are usually more accurate than the Field/Gallup/Rasmussen, etc. surveys. They tend to skew more Democrat and I wonder if that’s part of the reason why the GOP did better than expected.
If I can toot my horn, our surveys in California the absolute mean difference between our projections and the results was 3.9%. For most of the races we polled we were accurate within 2.5%. Where we were off the candidate out-performed our surveys. We saw a strong anti-incumbent sentiment that hurt both parties and helped challenger candidates who were outsiders.
But it does come back to assumptions of who will turnout to vote and how they will vote.
In a land of caller ID, call screening and voice mail, I’m surprised that political polling could be even remotely accurate. I never respond to polls or surveys. I’m sure I’m not alone. This has to skew the sample.
Leaders lead; they don’t follow polls. The only valid use of polling data is to see if the campaign is delivering its message effectively. The media love polls because it allows them to cover a sporting event rather than a contest of ideas and policies.
What you’re referring to is called non-response bias. There is some debate about whether the declining response bias affects accuracy as much as we’ve always thought. I believe there’s a few academic papers out there that show you can still get good numbers but that debate will continue.
Polling is not about finding out what to believe. You do it to develop the right strategy for success. It tells you how to leverage your resources to maximum effect. Not doing your research is like a general telling his troops to run in some direction with guns blazing. Real leaders do use research and the good ones know how to use it.
Polling = math = science
Behold, a god who bleeds.
From Peggy Noonan’s column in today’s WSJ:
This election produced lots of evidence on the con side of the debate.
Actually, this doesn’t really speak to non-response bias. This probably has more to do with whether the samples were representative (how many Reps/Dems? young people? Latinos?).