Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
Do We Overvalue Statistics? — Judith Levy
The fascinating website edge.org, which publishes well-considered reflections by smart people on very big questions, has made this its 2014 inquiry: What scientific idea is ready for retirement?
The respondents are well worth listening to: they include scientists of many stripes, mathematicians, philosophers, and economists, as well as several knowledgeable science writers and editors (and also, for reasons that are obscure, Alan Alda). The responses are posted one after another in a gigantic stream, forming a kind of alpha-listicle. It’s Buzzfeed for boffins, essentially.
A ton of big scientific ideas get voted off the island on the respondents’ page (see sampling below), but there is one response in particular that I thought might be of interest to our little subset, concerned as we are with things like elections. Emanuel Derman, professor of financial engineering at Columbia, wrote that the power of statistics is an idea worth retiring:
…nowadays the world, and especially the world of the social sciences, is increasingly in love with statistics and data science as a source of knowledge and truth itself. Some people have even claimed that computer-aided statistical analysis of patterns will replace our traditional methods of discovering the truth, not only in the social sciences and medicine, but in the natural sciences too.
… Statistics—the field itself—is a kind of Caliban, sired somewhere on an island in the region between mathematics and the natural sciences. It is neither purely a language nor purely a science of the natural world, but rather a collection of techniques to be applied, I believe, to test hypotheses. Statistics in isolation can seek only to find past tendencies and correlations, and assume that they will persist. But in a famous unattributed phrase, correlation is not causation.
Science is a battle to find causes and explanations amidst the confusion of data. Let us not get too enamored of data science, whose great triumphs so far are mainly in advertising and persuasion. Data alone has no voice. There is no “raw” data, as Kepler’s saga shows. Choosing what data to collect and how to think about it takes insight into the invisible; making good sense of the data collected requires the classic conservative methods: intuition, modeling, theorizing, and then, finally, statistics.
Bart Kosko, an information scientist and EE and law professor at USC, responded similarly that statistical independence is an illusion:
It is time for science to retire the fiction of statistical independence.
The world is massively interconnected through causal chains. Gravity alone causally connects all objects with mass. The world is even more massively correlated with itself. It is a truism that statistical correlation does not imply causality. But it is a mathematical fact that statistical independence implies no correlation at all. None. Yet events routinely correlate with one another. The whole focus of most big-data algorithms is to uncover just such correlations in ever larger data sets.
Statistical independence also underlies most modern statistical sampling techniques. It is often part of the very definition of a random sample. It underlies the old-school confidence intervals used in political polls and in some medical studies. It even underlies the distribution-free bootstraps or simulated data sets that increasingly replace those old-school techniques.
White noise is what statistical independence should sound like… Real noise samples are not independent. They correlate to some degree.
Science journalist Charles Seife wrote that “statistical significance” is almost invariably misused, to the point that it has become
a boon for the mediocre and for the credulous, for the dishonest and for the merely incompetent. It turns a meaningless result into something publishable, transforms a waste of time and effort into the raw fuel of scientific careers. It was designed to help researchers distinguish a real effect from a statistical fluke, but it has become a quantitative justification for dressing nonsense up in the mantle of respectability. And it’s the single biggest reason that most of the scientific and medical literature isn’t worth the paper it’s written on.
What say you, Ricochetti? First, what’s your take on the ever-growing popular reverence for statistics? And second, what scientific idea do you believe ought to be retired? To get your gears turning, here’s the promised selection of ideas the respondents came up with at edge.org (and check out the site; there are lots more where these came from):
- Mental illness is nothing but brain illness (Joel Gold and Ian Gold)
- Animal mindlessness (Kate Jeffery)
- Altruism (Jamil Zaki and Tor Norretranders)
- Entropy (Bruce Parker)
- Moral “blank slate-ism” (Kiley Hamlin)
- Natural selection is the only engine of evolution (Athena Vouloumanos)
- Opposites can’t both be right (Eldar Shafir)
- Left brain/right brain (Sarah-Jayne Blakemore and Stephen M. Kosslyn)
- The self (Bruce Hood)
- Beauty is in the eyes of the beholder (David M. Buss)
- String theory (Frank Tipler)
- Emotion is peripheral (Brian Knutson)
- Unification, or a theory of everything (Marcelo Gleiser, Geoffrey West)
- The intrinsic beauty and elegance of mathematics allows it to describe nature (Gregory Benford)
- Cause and effect (W. Daniel Hillis)
- Evidence-based medicine (Gary Klein)
- There can be no science of art (Jonathan Gottschall)
- Mouse models (Azra Raza, MD)
- The clinician’s law of parsimony (aka Occam’s Razor) (Gerald Smallberg, Jonathan Haidt)
- Standard deviation (Nassim Nicholas Taleb)
- Artificial intelligence (Roger Schank)
- Human nature (Peter Richerson)
- Programmers must have a background in calculus (Andrew Lih)
- Free will (Jerry Coyne)
- The Big Bang was the first moment of time (Lee Smolin)
- One genome per individual (Eric J. Topol, MD)
- Languages condition worldviews (John McWhorter)
- Infinity (Max Tegmark)
Totally agreed. But criticisms of a science shouldn’t come from our reading of the pop-literature interpretation of them.
Gödel’s Ghost, so again you respond by not actually responding. You give me examples where it is used (which I never claimed it’s not used), without addressing the question of…practically…what difference does it make to…most…methods. Saying that it has applications in some situations doesn’t address this either, as I already said that it has obvious superiority in some circumstances. The rest needs support in practice. Taking a Bayesian interpretation to get to the exact same position on methods we already use, has no practical significance, even if it has philosophical significance.
I’m not the one who’s arguing to throw away everything and replace with something else. Nor am I the one with the “extreme dogmatism”.
“Now, the criticism in the OP was that statistics in social sciences has problems. So far, I haven’t seen any serious arguments or evidence for this, despite 9 pages of conversation. And yes, I am also someone who makes a living on…applying…these tools…in academia”
Maybe that’s the difference. You’re in academia, I’m out in the business world where I see inappropriate uses of statistics all the time. Some management consultant goes into a business and declares that they have to start ‘putting some numbers’ behind what they are doing. So the brainstorming starts, and what often comes out is some kind of sampling regimen, such as surveying customers and feeding their responses into various algorithms designed to justify new product introductions, or to streamline customer service, or whatever.
Often these projects will result in reams of documents full of histograms, confidence intervals, impressive P values, etc. The management consultant’s job is done. In the meantime, no one noticed that the survey questions were horribly biased, or the effect of the wording was unknown, or the survey population was self-selecting, or whatever. Garbage in, highly numerical and ‘statistically significant’ garbage out.
Isn’t that…you job…to notice? This isn’t a criticism of statistics, or of social sciences, or of the application of statistics to social sciences. It certainly isn’t a criticism of what is published.
Also, don’t you think that there are obviously…good…applications of this? Companies like Nielsen don’t get paid millions of dollars a year by Kraft foods, for example, because their methods don’t work.
Also, what you describe ( poor application), is a criticism that can be applied to…everything, in every field. I was an engineer and I did SPC (just like you), and it’s done pretty poorly in many cases in engineering.
So my final conclusion is: these criticism can be applied to every field of science. And these criticism don’t imply “abandoning” something when you have no better alternative. The point of science is to improve its methods, which implies that they are not perfect.
I guess the conversation is following two different tracks. I am responding to the OP’s first question:
Jaynes alone addresses your points. The claim isn’t that Bayes lets you get the same results as orthodox statistics. It’s that when orthodox statistics is sound Bayes gives you the same results with less calculation; when orthodox statistics is unsound Bayes is sound; and Bayes lets you address problems orthodox statistics can’t even express.
As for your not having “extreme dogmatism,” I found it necessary to quote your earlier posts partially to show what a risible claim that is.
@#98
GG: …”The claim isn’t that Bayes lets you get the same results as orthodox statistics. It’s that when orthodox statistics is sound Bayes gives you the same results with less calculation; when orthodox statistics is unsound Bayes is sound; and Bayes lets you address problems orthodox statistics can’t even express.”
MA: Bayes demands a Prior assumption though, and, while showing the sensitivity of A Posteriori probabilities to different Priors is instructive, this methodology invites its own kind of misuse.
I was somewhat involved in such a case years ago:
The government wanted to purchase a certain number of test articles to prove out the reliability of the deployed article. Well, the mandated requirement on the demonstrated reliability and confidence level for that demonstrated value to come out of the test program was so high, based on classical statistics, that the government scrambled around for a way to justify a smaller, less costly test program, with fewer test articles needing to be purchased. This they accomplished by ‘inventing’ a Prior reliability that biased the test program toward’s higher reliability results, achieving the desired result in reduced test articles needed to confirm the reliability requirement value.
This is a ‘Classic’ example (excuse the pun) where Bayes methods can produce shady ways of doing business when unfettered from the constraints of classical structure.
And there you have it. What you said Gödel’s Ghost is actually…opinion. You still skirted my question of you showing how…practically…this has implications for most social science research.
Still nothing. But, then again, so is the nature of all these “Bayesian” discussions. Dogmatic “Bayesians” come in, declare everyone to be an idiot, and don’t feel the need to explain themselves. So what’s new?
Now, here’s the interesting thing: if the Bayesian assumption was so obviously superior, why aren’t social scientists falling over each other to adopt it in order to gain an advantage over their peers? I know! They’re all wrong.
The answer that a non-dogmatic person would give, is that obviously Bayesian has its superiority in certain applications, “frequentist” in others, and there’s also huge overlap between them. And the matter of more “efficient” or “philosophically correct” are both not only open to debate, but far from obvious.
But, that’s too much to ask for…
Well of course. I thought we were talking about inappropriate uses of statistics. And sure, there are misuses of statistics in every field, including engineering and the hard sciences. That’s not really in dispute. And of course,properly applied, statistical analysis is incredibly useful, including in the social sciences.
Does Judith agree that all investment/financial “quants” should be subject to very careful vetting before we accept any of their recommendations? Or does she still believe hubby? (grin)