The Poverty of Observational Data

A recent National Journal article does a good job of feeling out the limitations of traditional survey research and the modeling derived from it:

Overall, 2012 brought more attention than ever to poll aggregators, with their methods becoming more sophisticated. But where do they go from here?

“I don’t think there are great advances in averaging or modeling horse-race polling data,” Blumenthal said. “We are ultimately reliant on the quality of the data that’s collected.”

Blumenthal is correct; there is little more to be done with data from static snapshots of public opinion that traditional polling delivers. There is only so much information contained in any one poll or even large collection of polls. 

What ES does is add new information regarding how specific messages (information, ads, etc) push and pull opinion. It produces data from a counter-factual universe; what if every citizen watched David Gregory fiddle with a high-capacity magazine while poking at Wayne LaPierre? Would it shift opinion on gun control? Would LaPierre’s response move people? What about the impact of seeing both together?

This tells us how dynamic or stable opinion on a subject is, and where it is likely to move as an issue is engaged more seriously in the public sphere or in the media. Surveys and even tracking polls cannot accomplish this. Instead of just an X-Ray snapshot, we get something more akin to an MRI of political psychology, data which creates an image of how the application of a “treatment” affects a “patient.”

What made the Obama campaign so very accurate in their prediction of the vote across contested states was the use of experimental results from the “lab” and the “field” in their voter modeling. Because they had a large amount of experimental data showing them how different kinds of people shifted in response to various messages (toward or away from Obama, greater or lesser likelihood of voting), they could predict with astonishing accuracy the aggregate results.

Traditional polling data alone can never accomplish this, as the Romney Campaign’s shock at the outcome attests. This, despite the fact that they employed “vector autoregression models” to decipher the data and make their predictions. No matter what shiny statistical talisman one uses, none can overcome the inherent limitations of observational data. (More can be done with quasi-experimental designs such as regression-discontinuity, but these are more helpful in public policy than political psychology). 

So while there are likely no great advances to be made in traditional survey-based modeling, there are many exciting advances that have been made and yet to be accomplished using experimental data and creative research designs.

There’s a whole new world out there . . .

  1. Hang On

    There are an enormous number of ideas and at its heart are the ideas introduced in The Persuadable Voter, i.e., the voters most easily persuadable are not independents but partisans on the other side who disagree with the party about an issue of importance to them. It’s often written especially on this site that those who go back and forth are low-information voters when it’s far from the truth. It’s a matter of being partisans moving away and tugged back precisely because of issues. Use of the various techniques to identify and persuade are at the heart of a campaign and Obama did that well. There is a second element not identified in the literature you’ve cited and that is how to dissuade voters who might vote against you from showing up on election day and voting against you. I think Obama also did that effectively with his economic warfare message against Romney.

    Experimental_methods_can_be_used_both_to_identify messages and identify target audiences. But what about bias that might be introduced as a result. Alan Gerber is cited as being the author of article on New Haven’s turnout but he’s had bias problems in the past.

  2. Adam Schaeffer
    C

    Hang On . . . Agreed that persuading citizens to not vote for the other guy is an important and under-investigated topic. Certainly it seems this was a major problem for Romney. I think Republicans could do a lot of good, short and long-term, by reaching out to people who won’t vote Republican that election, but might be convinced that the Democrat isn’t worth their vote either.

    I think one problem is our collective fetish about voting; voting is considered inherently good, and the more voters the better. Anything that reduces turnout is considered inherently bad. Just look at Rove crying “vote suppression!” because Obama did a good job convincing a lot of white voters in Ohio that Romney is a cold-hearted Vulture Capitalist.

    Vote suppression means using tricks, fraud, and unfair barriers to prevent people from voting, not attacking another candidate. It’s just as repulsive to hear Rove make such a claim as it is for Dems to cry suppression because of voter ID.

    I think not voting is a perfectly reasonable and respectable choice. And it seems likely that not voting will be one step on the way to the other side for many voters.

  3. skoook

    Its a mistake to blend data mining with the simpler concept of tele-polling. The Obama campaign bulit a database from the ground up targeting Battleground states. They populated it with qualified voters and appended in excess of 100 data points on each voter. The database had architectural lifts from multi level marketing organizations. It used a combination  of the facebook/twitter concepts as a communication platform  and ” get out the vote”tool .

    They redefined the concept of ground game. Rance Priebus, take note  

  4. Duane Oyen

    I think that this stuff is OK at the margins as liong as you don’t overspend on it.  There are way too many people wasting time in Washington trying to be campaign experts.

    Ultimately, you win because you have the right message, not the cutest methodology.  And our message is stuck in 1986- cut taxes and stop the Commanists.

  5. Adam Schaeffer
    C
    skoook: Its a mistake to blend data mining with the simpler concept of tele-polling. The Obama campaign bulit a database from the ground up targeting Battleground states. They populated it with qualified voters and appended in excess of 100 data points on each voter. The database had architectural lifts from multi level marketing organizations. It used a combination  of the facebook/twitter concepts as a communication platform  and ” get out the vote”tool .

    Could you expand on this? What do you mean it’s a mistake to blend data mining and polling?

  6. Adam Schaeffer
    C
    Duane Oyen: I think that this stuff is OK at the margins as liong as you don’t overspend on it.  There are way too many people wasting time in Washington trying to be campaign experts.

    Ultimately, you win because you have the right message, not the cutest methodology.  And our message is stuck in 1986- cut taxes and stop the Commanists. · 18 hours ago

    I agree our messaging needs updating and different angles for different folks. But using the experimental method is how we will get to better, more effective messages. The experiment method is a tool, and can be used for improving agricultural yields, medicine, messaging, and campaign activities. And I do think that using that tool will be crucial to winning in the future . . .

  7. Duane Oyen
    Adam Schaeffer

    Duane Oyen: …………………

    Ultimately, you win because you have the right message, not the cutest methodology…. our message is stuck in 1986- cut taxes and stop the Commanists.

    I agree our messaging needs updating and different angles for different folks. But using the experimental method is how we will get to better, more effective messages. …….

    It’s not the messaging- that is my whole point.  We fool ourselves into thinking this is about techniques  and strategy.

    It is not.  Our philosophy and positions are stuck in an era that doesn’t exist any longer.  We need to learn that the market- the voting public- doesn’t care about small government, it cares about some kind of security in an uncertain world.   If we can’t figure out a way to address limited government, and markets, and freedom, as opposed to Ron Paul Small as our platform, we will go the way of the Whigs.

    This is about the message, not the method.  Obama did not win because he used clever Twitter techniques, he won because his message hit home to people who (rightly) felt that we don’t care about them.  That is the same issue with minorities.

Want to comment on stories like these? Become a member today!

You'll have access to:

  • All Ricochet articles, posts and podcasts.
  • The conversation amongst our members.
  • The opportunity share your Ricochet experiences.

Join Today!

Already a Member? Sign In