In (Weak) Defense of Nate Silver
Considering the number and intensity of the attacks conservative pundits have launched at New York Times election prognosticator Nate Silver, he's probably beginning to feel a bit like a piñata. I sympathize because -- even though I'm still recovering from injuries received defending John McCain (docs say the physical therapy is going great, by the way, thank you so much for asking) -- I'm now assuming the mantle of Only Conservative to Defend Nate Silver.
Not a whole-hearted defense, mind you, but one has to call ‘em as one sees ‘em, and, in my view, criticisms such as this one by Josh Jordan (nom de Twitter: @numbersmuncher), attacking Silver/Piñata’s methods and the conclusions based thereon really miss the point. But that said, Paul Krugman’s recent defense of Silver is equally off-point. Sorry, guys, but you’re both wrong, and for the same reason: confusing probability with prediction.
To understand the difference – and Jordan and Krugman’s error – let’s look at a proper, real-world application of probabilities: Suppose you are a purchasing manager receiving a shipment of 500,000 bearings. Suppose further that the measured size of 95 percent of the ball bearings must be within a tolerance of .001 inch. Should you accept the shipment?
Of course, you are not going to measure each one of 500,000 ball bearings. Instead, you pull a random sample of 20, 50, 100, or whatever (the more, the better) bearings and measure each one. If they are all within tolerance, great. But what if one bearing out of, say, 50, is a outside of tolerance?
That’s where probability saves the day: Using a simple formula, it is possible to calculate the probability that one bearing out of a random sample of 50 would be out of tolerance from a sample of 500,000 of which 95% should be within tolerance. If the probability is high, you accept the shipment.
But – and this is the important point – a high probability is not a guarantee. When, ultimately, every bearing is measured before installation, it is still possible that more than five percent of the bearings will be bad. But you – and this is the very important point – would keep your job.
Which brings us to Nate Silver and what is important to understand about his election model, which is this: When Silver writes, say, that his model gives Barack Obama, say, a 70 percent chance of winning the 2012 election, what he is saying, in fact, is that if we could hold, say, 100 identical presidential elections with the same candidates under identical circumstances, Obama should win, on average, 70% of them.
But of course, that’s impossible and if you’ve been following my explanation, you understand that one election is a ridiculously small sample. If Obama loses, Silver’s response would simply be that this was one of the three-out-of-ten instances where Silver’s model “predicted” an Obama loss. That’s what is, in this writer’s view, so useless about the attacks on Silver: No matter who wins the election, there is no definitive way to prove Silver’s model wrong.
The above is the long explanation. Here’s the short one: The day before the Supreme Court issued its Obamacare ruling, the political betting website Intrade showed the probability of the Court striking down the law at 75.5 percent.
- Comment (18)
- · Quote
- · UnfollowFollow (3)
Comments:
Jul '12
Re: In (Weak) Defense of Nate Silver
Welcome aboard. "Defending" Nate Silver in your first post is definitely gutsy. If I may channel Biden, Have you always had ball bearings the size of cue balls?
Edited on November 6, 2012 at 12:58amIt's an insightful peice. I look forward to more.
Mar '11
Re: In (Weak) Defense of Nate Silver
It takes a brave man to defend Nate Silver on a conservative website the day before the election. Kudos.
Gene Schwimmer, Guest Contributor:
If Obama loses, Silver’s response would simply be that this was one of the three-out-of-ten instances where Silver’s model “predicted” an Obama loss. That’s what is, in this writer’s view, so useless about the attacks on Silver: No matter who wins the election, there is no definitive way to prove Silver’s model wrong.
Of course, this same observation also makes all of Silver's analyses themselves useless.
Newspaper readers and pundits don't want and can't deal with probabilities. They want predictions: who will win, who will lose.
Nate Silver and the NYT have no one to blame but themselves if they did not realize that neither the elite punditry nor the unwashed masses is capable of rationally interpreting probabilities. His analysis is implicitly sold as an up/down prediction, and he shouldn't be surprised if that is the petard he is hoisted by after the election.
Edited on November 6, 2012 at 1:03amAug '10
Re: In (Weak) Defense of Nate Silver
Nate Silver's explanation in his Nov 2nd post seems a fair enough self-defense: For Romney to Win, State Polls Must Be Statistically Biased
In which he explains probability as it relates to his model:
So Silver's percentages represent discrete possibilities, and 16% is somehow the possibility all polls are biased.
Of course, as you point out, when dealing with probabilities it is very hard to falsify the model; post-hoc analysis can just claim that "something changed". Which is why probability models are problematic for prediction purposes.
Edited on November 6, 2012 at 1:25amJun '11
Re: In (Weak) Defense of Nate Silver
Welcome! There's been a small bit of discussion of Nate Silver over on the member side. I'm not a reader of Nate Silver's column but became aware of it through references to it by traders on Intrade. One of the things that makes predictive markets accurate is the lack of communication between traders. I became aware that lots of traders were following Silver's models and participating in groupthink. That can damage the wisdom of crowds. Which is why I shorted Obama futures.
Perhaps a minor point, but...
The issue traded on Intrade involving Obamacare was not that the Supreme Court would "strike down the law". It was: "The US Supreme Court to rule the individual mandate unconstitutional before midnight Eastern Time 31 Dec 2012." (There's a second one dated 2013.) Both are still open issues.
It's clear that Intrade will consider the mandate to have been ruled unconstitutional but many traders have vigorously and stubbornly pointed out that SCOTUS actually didn't-- they changed it to a tax not a mandate and ruled the tax to be constitutional. Which is one reason why neither issue has settled at zero to this day.
Aug '10
Re: In (Weak) Defense of Nate Silver
Interestingly, given Nate Silver's own words above, if Romney wins then Silver must conclude that the polls were all biased against him. He must then junk his model, because it treats polls as raw data but models built with bad data are flawed. Or he could claim to be able to measure that bias in such a way as to weight polls more accurately.
I don't think Silver does bad work--he certainly seems to have a solid grasp of the basics--and I am genuinely interested to hear what his analysis would be on the day after a Romney win.
There were cases of similar popular models that had to be re-thought. The guy who runs electoral-vote.com got the 2004 call wrong, and engaged in much navel gazing and mumbling afterwards. As someone who obsessed over his maps in the months leading up to it, reading those posts were wonderful entertainment.
(Interestingly, he now offers "Rasmussen-free maps", based on Nate Silver's analysis claiming that Rasmussen polling is biased towards the GOP. The irony is that, if he had excluded notoriously biased Zogby polls in 2004, he would have got more accurate results.)
Apr '11
Re: In (Weak) Defense of Nate Silver
This is why I have come to find polls worthless. I don't think I have ever read about so many polls in my short life as I did this year, and I am utterly convinced there is no real benefit to the public to either analyze or discuss these things.
The fixation on polls is a media construct, which requires a graphical representation of the election contest. Polls provided something to track over the months of this terrible process, that would give it a sports like narrative that News fans could relate to.
May '12
Re: In (Weak) Defense of Nate Silver
While I agree with the overall spirit of your defense, election models and sampling of ball bearings aren't similar. First off, a 95% probability for ball bearings gives you a ppm of 50,000, which is probably much more horrible than any Soviet Gulag factory ever produced, by several orders of magnitude. Second, sampling of ball bearings and people are different, in that there is no determinism involved with ball bearings. Randomness is a sufficient guarantee. It is never so simple with people, especially taking into account the variables one uses in the model, and the fact that the n of presidential races to test the model is very small (and the fact that there is never randomness)
So, while Nate Silver may be doing the "right" thing, statistically speaking, the models can never be used to predict. Should, I should say. Some people misunderstand this and think that since the statistics are fine, than the model is a reliable prediction tool. There's no such thing as a predictive model in social sciences (shouldn't I should say)
Also, there is no doubt that state polls are statistically biased. That's the whole point of house effects.
Mar '12
Re: In (Weak) Defense of Nate Silver
Welcome. I will quote my similar in spirit (but far more stripped down) defense of Silver a few days ago on Daily Caller:
Guy comes up to bat with bases loaded. Silver in the seat next to you says, "Guy's got a 70% chance of striking or flying out." Guy gets a hit, and Silver says, "Good player to have in there in this situation, a .300 hitter." And, he's right.
Jul '12
Re: In (Weak) Defense of Nate Silver
BlueAnt: Nate Silver's explanation in his Nov 2nd post seems a fair enough self-defense: For Romney to Win, State Polls Must Be Statistically Biased
In which he explains probability as it relates to his model:
Edited 36 minutes ago
I bolded the part where Silver is editorializing. The statistical bias argument is much more than wishful thinking, and I think he knows it (unless he's in complete denial). But his post is a nice, clean way of weaseling out of the mess he's created when Romney wins.
Aug '10
Re: In (Weak) Defense of Nate Silver
Recovering Liberal
I bolded the part where Silver is editorializing. The statistical bias argument is much more than wishful thinking, and I think he knows it (unless he's in complete denial).
Yes, that's the clear weak point in his analysis. But to be fair, if one believes in a statistical model--whether scientist, economist, or columnist--he must believe in the core integrity of the data. I view this less as a personal bias and more of a professional bias: a data wonk just can't believe that all the data is bad, because that erases his reason for being.
In fact, for a weak point, it's fairly strong. I obsess over poll internals as well, and while I see repeated evidence for bias I have a hard time believing all polls are biased.
Citing 16% as the probability of complete systemic error is not unreasonable. But it does avoid the larger question: the need to examine the whole enterprise of model-making. You won't convince a numbers guy that he can't measure something, any more than you will convince a macro-oriented Keynesian that he can't model a complex economy using aggregates.
Apr '11
Re: In (Weak) Defense of Nate Silver
I predicted Super Tuesday a little better than Silver. I'm proud of that. Silver's model is composed of polls and fudging that he puts in place well ahead of time (plus a little custom fudging in poll weighting). I don't pretend that I could beat him regularly.
I'm not proud of doing better than Sean Trende, who was wrong about race after race after race, who couldn't even handle the math of the outcomes of his predictions. It turns out that prediction is hard, it's hard to see your biases, and Trende was unlucky enough to have biases aim in the wrong direction. I suspect that Silver's partly in that place now.
My chief hopes for this Silvers' error are that Mitt has moved substantial funding from advertizing to GOTV. The result that you would expect from that is that his poll results would be less good, but his performance relative to the polls would be better.
What I had hoped for was that we'd have better early voting results, but we appear not to have gotten the volunteers we needed a month ago. I still hope we'll outperform tomorrow.
Jul '12
Re: In (Weak) Defense of Nate Silver
Further to AIG's point, to appreciate the difference between assessing probabilities involving ball bearings and predicting elections involving human voters, one would have to imagine the challenge of the former if some ball bearings actively avoided being measured (like people who refuse to answer telephone surveys), if some ball bearings could change their size between the sample measurement and installation, or if some ball bearings were one size, but when measured appeared to be a different size.
Apr '11
Re: In (Weak) Defense of Nate Silver
Looking at Silver's prediction right now, he has Mitt down to a 7.8% chance. That's not falsifiable, but it might as well be. My position earlier was that Mitt could win narrowly without humiliating Silver. I withdraw from that position.
Silver has him losing Colorado at an 81% chance. He's ahead in early voting there. Colorado's much easier for conservatives to bus into than for liberals. It's a state with a non-trivial LDS population. 81% seems very high indeed. If Mitt doesn't lose by a substantial margin in Colorado, I will lose a substantial margin of respect for Silver.
Mar '11
Re: In (Weak) Defense of Nate Silver
Hey nobody said we are going to have to do math on this website. Math is hard.
Re: In (Weak) Defense of Nate Silver
I'm with you, Gene. Nate Silver's probably mistaken, but he lays out his analyses in a fairly transparent way, enabling readers to judge for themselves. I've learned a lot--and I mean a lot--by reading him.
Oh, and Gene? One more thing.
Welcome to Ricochet.
Sep '12
Re: In (Weak) Defense of Nate Silver
I think this defense of Silver (and I've seen it in a couple of contexts) is fundamentally flawed. The striking difference is that in your example, the ball bearings are real. The truth of their precision and form exists independently of your measurement, and probability is a way to try to get at that truth.
In Silver's world, though, the simulations he runs are not real. There are not, in fact, 100 distinct presidential elections we're working through, and so there is no underlying "truth" these models are actually uncovering. All they are doing is reflecting their inputs. As such, the only real utility of Silver's endeavor is whatever predictive value it has.
Thus, a Romney victory absolutely destroys Silver's position. If you project a 90% probability based on your theoretical models and are wrong, it is insufficient to just say, "well, that was the 1 in 10 case." If your model's only real value is tracking future events and you lose out on that, what good is it? (Global warming comes to mind.) So I don't think the probability/prediction distinction does the work you need it to here.
May '12
Re: In (Weak) Defense of Nate Silver
What Silver is doing statistically is just fine. There's no reason to criticize the technical details. However, this doesn't mean that the model has face validity, or that any of these models have face validity. These models require some assumptions, since one can never randomly sample voters. That's why they do things like proportions, but to come up with those, one has to make some assumptions.
So the problems with these models, is the problem of all models in social sciences; given enough variables you can get any sort of statistical significance you want.
So one can neither criticize Silver, or hold his models as an assurance of the future. One can necessarily improve upon the model, but the model may not have any face validity.
Re: In (Weak) Defense of Nate Silver
Having just joined, it's taken me a couple of days to familiarize myself with the site, start navigating, etc., so excuse my tardiness in responding to the comments. But especially, thanks for the warm welcome! And the comments, whether they agree with me or not, are very good, which is to say that they are on point, inform and even enhance by adding new insights and angles.
Thanks, again, for the warm welcome. Hope the powers-that-be let me stick around!