Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
Who Reviews the Reviewers?
In an era where politicians and judges increasingly turn to academic research for information about issues that affect us all, “Is this science any good?” is, literally, not just an academic question. As I wrote about in The Federalist, The Unskewed Project — of which I’m the content editor — launched earlier this year to provide an extra layer of review on recently-published social science and on the reporting about it.
Unfortunately […] social science, exists in a highly politicized and media-saturated environment that celebrates novelty over consensus and drama over diligence. When presented with university press releases making bold claims, harried reporters often neglect to ask the kind of informed, challenging, and skeptical questions they’d ask anyone else. Even when journalists intend well, research that confirms popular worldviews tends to be welcomed without reflection. At the same time, scholars whose conclusions are out of fashion are often subjected to hostile scrutiny based less on evidence than ideology…
“[We] felt there was a need to provide an extra critique for research that was impacting the public dialogue” said Brian J. Willoughby, the editor-in-chief of Unskewed and an associate professor in the School of Family Life at Brigham Young University. “The echo chambers of the media and academia have led to a situation where research findings are less scrutinized and more blindly accepted more than ever before. Unskewed was created to provide a counter-point to flawed social science research being disseminated in the public.”
The piece summarizes a few of the more outrageous papers Unskewed has reviewed in the last few months, including articles on the (supposed) need to have fewer kids to stop global warming, the (presumed) crisis of access to abortion in red states, the (alleged) strengths of polyamory, and how even good science can get hijacked by bad journalism.
One of the things I didn’t get a chance to talk about much in that article is why this kind of research is passing the peer review process. Part of the answer is the effect the digital age has had on all media: The same forces that allow lone geniuses to directly reach mass audiences also apply to charlatans and cranks. The last few decades have seen an explosion in the number of published journals — Elsevier counts 23,700 and I’ve seen estimates that go even higher — and many of them are, frankly, junk journals that will take nearly anything. Papers that would never see the light of day 30 years ago now get published online where they can be pushed in unscrupulous press releases and picked-up by credulous journalists looking for a good story.
Another reason is that the reviewers aren’t always as skeptical as they should be. Sometimes, the error is egregious: One paper we reviewed was premised, in part, on the idea that we stop catastrophic climate change in the next 30 years by having fewer descendants over the next several-hundred years. Somehow, this tiny incongruity escaped the notice of the reviewers (for what little it’s worth, this was a paper on climate change education, not climate change itself).
Other times, it’s more about using science as a form of political advocacy. A paper Mark Regnerus reviewed for us was a survey of women who had to travel, either out-of-state or more than 100 miles, to procure an abortion. Its conclusions are fairly modest and almost mundane. The problem is that the question guaranteed only one possible answer: Of course, women who had to travel moderate distances to procure an abortion were somewhat inconvenienced by the travel. As for why the researchers bother to conduct — and the paper publish — such a paper, Regnerus thinks it’s plain:
I suspect this study is in print in order to reinforce “the literature,” that pool of studies that can and is used as a political and legal hammer when mustered. […T]his study seems tailored to help buttress legal issues around abortion access, and to help the Supreme Court’s decision in Whole Women’s Heath v. Hellerstedt remain in place. […] The more papers like this that] appear in peer-reviewed journals — genuine quality, novelty, and evidence for its conclusions be damned — the better, they surmise.
Science is too important to be misused like this. Let’s not let them get away with it.
Published in Science & Technology
I think the problem of junk science would be less of a problem if journalists were not so scientifically illiterate to begin with. There is an almost inexhaustible reservoir of credulity when it comes to “Studies show…” journalism.
Is this tied at all to the Replication Project?
It’s a neat way to build up a “97% consensus”.
If you generate a whole lot of papers on relatively minor or trivial questions, but which include a sentence or two identifying the paper as supporting the “consensus” position, it gets included into the pool of papers which support that “consensus”.
Neat trick.
Journalists can be scientifically literate and still be inexhaustibly credulous. They’re the ones, after all, who get to pick and choose which studies on which to report.
It is, especially as each individual paper is — on its own — perfectly valid.
Both problems are real.
On the one hand, you have reporters who will dutifully any college press-release that says “Science!” in it.
On the other… well, we’re all susceptible to stuff that looks like it confirms our own biases.
One other thing I meant to make more explicit: Peer review works best when there’s an adversarial relationship between the authors and the reviewers.
That is, authors should feel like prosecutors, in that the burden of proof is on them to make their case. Likewise, reviewers should be like defense counsel, looking for problems and weaknesses in the hypothesis presented to them. Their job isn’t to make a case of their own, but to disprove or cast doubt on the authors.
When journals are so niche that there aren’t potential points for controversy among the authors and reviewers, you’re more likely to get political advocacy with statistics rather than actual science.
Not directly, but of similar spirit.
I have written one peer reviewed article. I was challenging an article published in the same journal two years earlier. It would not surprise me if both articles had the same peer reviewer. Implicitly, I was arguing that the earlier peer reviewer was incompetent. After some hostility from the peer reviewer, my article was published without any significant changes.
That’s a big ask if the journal in question needs material in order to publish on deadline. They can’t be saying “no” so often that there’s nothing available to publish.
True.
It’s another consequence of proliferation and low-barriers to entry.
When there’s only a relative handful of publications, they get to be choosy and discriminating. On the plus side, that means they can enforce quality control and publish only the best. On the down side, it means lots of good stuff doesn’t see the light of day, or gets quashed because it doesn’t tow the line.
When there are few barriers to entry, you get more variety and dissenting opinions have a better chance of being heard. On the down side, you get a lot of schlock.
Great concept. I have been an amateur “study shows” critic for decades. I’m excited to see your content.
Is the logo intentionally designed to look like a riff on Vox?
Edit: By the way, I am confident the content on Unskewed will be less frivolous than the histories of feuds between rappers I’ve never heard of.
Thanks for this public service, Tom. When you’ve mastered the social science realm you should take on the hard sciences as well. The leftist infection is rampant (politicized science as agenda-driven, not truth-seeking).