The Problem with Field Experiments

Field experiments are spectacular tools for discovering what works in the mess of the real world at a given point in time. But in many ways, because of that mess of reality, they are often blunt and inefficient tools for investigating message effectiveness and political psychology in general.

A fascinating recent field experiment testing the political effectiveness of Facebook ads illustrates many of the problems of this approach.

A big problem with field tests is that they typically cannot disentangle who is actually exposed to a treatment and who is not. Field tests entangle mode of delivery with the message content, and therefore seriously degrade one’s ability to identify differences in message effectiveness. Essentially, you’re confounding tests of mode of delivery and message content. 

An example might help . . .

Joe is in the treatment group, and his Facebook page loads the ad treatment 10 times over three days (or he was left a flier at his house), but Joe might not have actually seen the ad (or flier). Maybe Joe has become very good at screening out these ubiquitous ads so that they don’t register even for the most fleeting moments (or perhaps Joe’s wife threw away the flier before he saw it). Joe was never meaningfully treated with the message because the mode of delivery was extremely ineffective for him.

Paul, on the other hand, tends to keep an eye out for Facebook ads for deals on his home beer-brewing hobby. Paul briefly scans the ads, and does see, and register fairly effectively, the ads on his page (or perhaps his wife hands him the flier to read). Paul was meaningfully treated with the message because the mode of delivery was relatively effective for him.

Both Joe and Paul are in the treatment groups, but only Paul was exposed to the treatment in a meaningful sense. And yet the analysis of the ad effects must include both Joe and Paul as being treated (this is called “intent to treat” analysis) because we can’t know who was effectively exposed to the ad (self-reports of exposure are notoriously problematic). 

Now, for a statistical test of the ad impacts, all the “Joes” and all the “Pauls” are lumped together in the “treatment” groups and compared to the control group. What that means is that any impact from being exposed to the message content will be diluted by the fact that none of the “Joes” were really exposed to the message. 

In the case of the Facebook experiment, they tested 3 different messages:

The first ad merely sought to build the candidates’ name recognition and identify him as a proud resident of the area:

[Name of candidate] for [Name of office]

My family is one of few younger families to move to [region]. Find out why!

[Picture of candidate with his family]

A second ad included a more explicit character appeal stressing the candidates’ business experience and military service:

[Name of candidate] for [Name of office]

I spent 12 years in [branch of the military] and grew a [region] small business. Connect with me today!

[Picture of candidate smiling and holding his campaign sign]

Finally, a third appeal sought to appeal to voters on a salient policy issue by stressing the candidates’ desire to improve farming in the state:

[Name of candidate] for [Name of office]

Farming is crucial to [state]’s economy. [Candidate’s first name]’s 4 WAYS to improve farming in [state] today!”

[Picture of candidate dressed nicely and giving a speech to a small crowd]

The candidate’s constituency includes a large number of people connected to the farming industry, and thus the candidate expected this to be a particularly salient issue.

The study found no significant effects from “intention to treat” voters with these Facebook ads, and no differences in effectiveness between the messages. 

Here’s the problem . . . we have no way of knowing wether these ads were ineffective because the mode of delivery is ineffective, the messages are all ineffective, or both. We have no way of knowing whether message 1, 2, or 3 is more effective if a voter is actually exposed to the message, not simply intended to be exposed. The field test most likely wasted a test of message content effectiveness by choosing a mode that was so ineffective.

In contrast, if voters were randomly assigned to these three messages within an online survey experiment, we can ensure to a much, much greater extent that Joe and Paul are both meaningfully exposed to the messages. We can prevent them from advancing in the survey until a certain amount of time has passed, we can ask them a question about the content, or we can instruct them to pay attention in advance. 

These are all artificial aspects of message delivery compared with the real world, but they ensure that we can identify the actual effect of being meaningfully exposed to a message. We can also test for specific mode/message effects and interactions . . . is a particular message more or less effective when delivered as an audio-only, radio ad, TV/web ad, or print flier? How do specific visuals moderate the impacts?

Furthermore, we can identify the impact of messages on subgroups that might be very difficult to reach through specific modes of delivery. 

All methods have pros and cons, but the much-greater control that we have over “lab” experiments offers a much more efficient means of investigating political psychology.

  1. Brian Clendinen

    However, from a Cost Benefit analysis standpoint, knowing how often a person actually pays  attention to the ad is also critical.

    If I have a message that is twice as effective when one actually pays attention to it than another,however half the number of people actually pay attention to it, then both are the same (assuming equal time exposed and placement cost rates are the same).

    Also another really import variable is the mode mix for print/web. The other modes surrounding an individual ad are as important as the individual messages mode.  

    Add into the equation demising scales of return from multiple exposers and how the effective  the messages are in causing a tipping point  to vote for your candidate; you still have a long ways to go before we can actually measure effectiveness to any sort metric(s) that are actually meaningful. Don’t get me wrong separating the two is a good next step and it is better to have good data separating one variable out of your underlying data.

    You need this level of accurcy to understand what the best medium mix is for a given budget.

  2. Adam Schaeffer
    C

    Brian . . . definitely agree . . . we need a mix of approaches, preferably integrated with one another in a thoughtful way to leverage the advantages of each and fill the weaknesses of each.

  3. Adam Schaeffer
    C
    Pilli: Was Marshall McLuhan right?  The medium IS the message?

    Sounds like that is the effect being studied if not intentionally then in actuality. · 1 hour ago

    Interesting . . . had to Google that reference! Have you read much of him/this area?

  4. Dave

    When I would close the Obama ads that popped up on Facebook, I got a small satisfaction from choosing “misleading” as the reason I didn’t like them.

  5. Brian Clendinen
    Adam Schaeffer

    Pilli: Was Marshall McLuhan right?  The medium IS the message?· 1 hour ago

    Interesting . . . had to Google that reference! Have you read much of him/this area? · 39 minutes ago

    I love McLuhan, however he is hard to read at times. I really wish someone would take his greatest work  ,4 laws of media, and write a more accessable book. His 4 laws of Media should be a standared in any marketing or communication texbook/primer.

     The problem was he died before he could write a book on the 4 laws. His son instead wrote the book and boched it. Even in the preface his son admits others read the book before  publishing and complained about the son trying to be creative in how he wrote the book which did not work and was confusing. He then bascily dismisses the critics and says your wrong. His freind were 100% right his book was chaotic not creative. Therefore his fathers 4 laws were ignored by everyone except  some acedemics.  

    I was actually first  exposed to McLuhan  via a sermon. McLuhan has concepts that, completely unintentionally, are very scriptural.    

  6. Misthiocracy
    Adam Schaeffer

    Pilli: Was Marshall McLuhan right?  The medium IS the message?

    Sounds like that is the effect being studied if not intentionally then in actuality.

    Interesting . . . had to Google that reference! Have you read much of him/this area?

    He’s generally required reading for us Communications Studies grads.

  7. Misthiocracy
    Brian Clendinen

    I was actually first  exposed to McLuhan  via a sermon. McLuhan has concepts that, completely unintentionally, are very scriptural.

    Not entirely unintentional.  He was a devout Catholic.

    He always said that he didn’t mix religion with academics, but he really had a very conservative attitude regarding the direction of pop culture and mass media during his lifetime.

    You see that reflected in most Comm Studies program. Politically the profs are very left-wing, and yet the attitude of “everything is crap nowadays” that permeates it feels like it’s coming from Clint Eastwood’s character in Gran Torino.

    However, that mostly an judgement of aesthetics. A judgement of the content, rather than the media, ironically. His analysis of the structures of media is much less judgmentally political.

    At least, that’s my opinion.

  8. Misthiocracy

    Apropos of nothing, but I prefer one of his lesser-known axioms: The content is the audience.

    I also really like one of his spiritual successors: Neil Postman.  Postman’s stuff is a bit easier to read. 

    I think kids at U of T gravitated to McLuhen at least in part because they could never really keep track of what the heck he was talking about. He was like a mystic, spilling out a stream-of-consciousness, and all without the use of psychotropic drugs.

  9. Pilli
    Misthiocracy

    Adam Schaeffer

    Pilli: Was Marshall McLuhan right?  The medium IS the message?

    Sounds like that is the effect being studied if not intentionally then in actuality.

    Interesting . . . had to Google that reference! Have you read much of him/this area?

    He’s generally required reading for us Communications Studies grads. · 29 minutes ago

    As a matter of fact, I was a Comm Studies student at U of T (Tennessee) in the late ’60s.  Misthiocracy, you are very perceptive.

  10. Pseudodionysius

    In this previously unpublished work, a young Marshall McLuhan, as cultural historian, illuminates the complexities of the classical trivium, provides the first ever close reading of the enigmatic Elizabethan writer Thomas Nashe, and implicitly challenges the reader to accept a new blueprint for literary education. Ideas that would ground McLuhan’s media analysis of the 1960s and 70s are here in embryo, as he sets out in scrupulous detail the role of grammar (interpretation), dialectic, and rhetoric in classical learning. Under McLuhan’s scholarly microscope, the internal dynamics of the trivium and its purpose are revealed. As is its indispensable role in giving full due to the rich prose of Thomas Nashe.

    In ranging over literature from Cicero to the sixteenth century, McLuhan discovers the source and significance of multiple traditions in Nashe s writings. Here, more than half a century after it was written, is a fresh, insightful, and richly coherent framework for studying Nashe and an unequivocal call for a program of education based on the ambitious and lofty ideal of reintegrating the classical trivium.

    Trivium.jpg

  11. Pilli

    Was Marshall McLuhan right?  The medium IS the message?

    Sounds like that is the effect being studied if not intentionally then in actuality.