Pinning the Bogometer, or Fake but Accurate in Psychology Research

 

The Worm Runners Digest was a journal of the 1950s and 1960s that published both satirical scientific articles and actual science. Wikipedia tells us that:

After complaints that the satirical articles and the scientific publications were not distinguishable, the satirical articles were printed upside down in the back half of the W.R.D. along with a topsy turvy back cover.

The Journal of Polymorphous Perversity ran to 40 issues over 20 years, and important articles are available in various anthologies. These include the first article, “Psychotherapy of the Dead,” and the perennial favorite, “Oral Sadism and the Vegetarian Personality.”

The Journal of Irreproducible Results has been appearing under changing management for over half a century now.

Unfortunately, life seems to be imitating art these days, as landmark papers in psychology turn out to be difficult or impossible to reproduce.

Landmark, you ask? The Stanford Prison Experiment for one. You know, the one where undergraduates randomly assigned to be “prison guards” had the power go to their heads and became sadistic monsters. Or not. From early in the study’s career, there was criticism:

Despite the Stanford prison experiment’s canonical status in intro psych classes around the country today, methodological criticism of it was swift and widespread in the years after it was conducted. Deviating from scientific protocol, Zimbardo and his students had published their first article about the experiment not in an academic journal of psychology but in The New York Times Magazine, sidestepping the usual peer review. Famed psychologist Erich Fromm, unaware that guards had been explicitly instructed to be “tough,” nonetheless opined that in light of the obvious pressures to abuse, what was most surprising about the experiment was how few guards did. “The authors believe it proves that the situation alone can within a few days transform normal people into abject, submissive individuals or into ruthless sadists,” Fromm wrote. “It seems to me that the experiment proves, if anything, rather the contrary.” Some scholars have argued that it wasn’t an experiment at all. Leon Festinger, the psychologist who pioneered the concept of cognitive dissonance, dismissed it as a “happening.”

That’s from a Medium mag expose of the “study.” I think after you read the article you’ll see that the scare quotes are appropriate.

Here’s another tidbit:

…the tale of guards run amok and terrified prisoners breaking down one by one has become world-famous, a cultural touchstone that’s been the subject of books, documentaries, and feature films — even an episode of Veronica Mars.

The SPE is often used to teach the lesson that our behavior is profoundly affected by the social roles and situations in which we find ourselves. But its deeper, more disturbing implication is that we all have a wellspring of potential sadism lurking within us, waiting to be tapped by circumstance. It has been invoked to explain the massacre at My Lai during the Vietnam War, the Armenian genocide, and the horrors of the Holocaust. And the ultimate symbol of the agony that man helplessly inflicts on his brother is [Douglas] Korpi’s famous breakdown, set off after only 36 hours by the cruelty of his peers.

There’s just one problem: Korpi’s breakdown was a sham.

Then there is the famed marshmallow test. You know. The one where kids who had the willpower to wait to eat a marshmallow turned out to do better in school.

So, as part of the long overdue project of trying to fix psychology’s “replication crisis,” a project trying to reproduce the work in seminal papers is underway. The marshmallow test made the list. Here’s what happened:

The idea behind the new paper was to see if research from the late 1980s and early ’90s showing that a simple delay of gratification (eating a marshmallow) at ages 4 through 6 could predict future achievement in school and life could be replicated.

What the researchers found: Delaying gratification at age 5 doesn’t say much about your future. Rather, there are more important — and frustratingly stubborn — forces at work that push or pull us from our greatest potential.

The marshmallow test story is important. The original studies inspired a surge in research into how character traits could influence educational outcomes (think grit and growth mindset). They also influenced schools to teach delaying gratification as part of “character education” programs.

Oops.

The New York Times published its article under the title:

Psychology Itself Is Under Scrutiny. Many famous studies of human behavior cannot be reproduced: Even so, they revealed aspects of our inner lives that feel true.

Glenn Reynolds asked:

How much of the “settled science” will turn out to be a pack of politicized lies?

Since the NYT’s headline reads, “The studies cannot be reproduced: Even so, they revealed aspects of our inner lives that feel true,” he answers his own question:

…enough that they’re already retreating to the “fake but accurate” defense.

Published in Science & Technology
This post was promoted to the Main Feed by a Ricochet Editor at the recommendation of Ricochet members. Like this post? Want to comment? Join Ricochet’s community of conservatives and be part of the conversation. Join Ricochet for Free.

There are 7 comments.

Become a member to join the conversation. Or sign in if you're already a member.
  1. TRibbey Inactive
    TRibbey
    @TRibbey

    Ontheleftcoast:

    Unfortunately, life seems to be imitating art these days, as landmark papers in psychology turn out to be difficult or impossible to reproduce.

    Landmark, you ask? The Stanford Prison Experiment for one. You know, the one where undergraduates randomly assigned to be “prison guards” had the power go to their heads and became sadistic monsters. Or not.

    Yes! I heard about this and was not surprised for some reason. I am going to see Jordan Peterson in October and will submit a question in the Q&A about the SPE to see if he has any thoughts on the matter.

    • #1
  2. RightAngles Member
    RightAngles
    @RightAngles

    Yep, the words “Studies show…” are often followed by whatever the speaker wants you to believe.

    • #2
  3. Paul Erickson Inactive
    Paul Erickson
    @PaulErickson

    RightAngles (View Comment):

    Yep, the words “Studies show…” are often followed by whatever the speaker wants you to believe.

    And when the replication project is complete, they can say, “Studies show that the words ‘Studies show…’ are often followed by whatever the speaker wants you to believe.”

    • #3
  4. Hank Rhody, Possibly Mad Contributor
    Hank Rhody, Possibly Mad
    @HankRhody

    I’d like to see this sort of checking done on a heck of a lot more subjects than Psychology.

    If you’re doing rigorous, double blinded testing on medicine, for example, and you get 95% confidence in your results, then out of twenty papers you can expect one of them to be wrong, even if everything else is done perfectly.

    • #4
  5. Ontheleftcoast Inactive
    Ontheleftcoast
    @Ontheleftcoast

    Hank Rhody, Possibly Mad (View Comment):

    I’d like to see this sort of checking done on a heck of a lot more subjects than Psychology.

    There’s doing it, and then there’s publishing it. There’s strong reason to think that negative studies that would kill a pharma goose that lays a golden drug sometimes don’t see the light of day. Even if there are more of them than the ones that polish the gold

    • #5
  6. TBA Coolidge
    TBA
    @RobtGilsdorf

    I pose the idea that any study that hasn’t been replicated be dismissed out of hand when public policy is being debated. 

    • #6
  7. John H. Member
    John H.
    @JohnH

    If among the “frustratingly stubborn” forces there is people’s own DNA, then I don’t see these dusty psychological treasures losing value. Being disproven is not the same as being discredited.

    • #7
Become a member to join the conversation. Or sign in if you're already a member.