Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
Coronavirus: About Those IHME Projections…
As most of you know, the Institute for Health Metrics and Evaluation (IHME) at the University of Washington has published a series of COVID-19 projections over the past couple of weeks. The IHME projections are sometimes called the “Chris Murray” model, after its director. It appears that these projections are being widely used by the Trump administration and other government officials. The main site is here.
The IHME released its original projections on March 26, and has released updates on March 30 and 31 and April 1, 2, 5, and 7. It appears that only the most recent projection is available for download at the site. From previous downloads, I have all of them except April 1 (which appears quite similar to April 2, based on the update notes that are available for each update here).
This post is an evaluation of the changes in the IHME projections. I focus on two issues: (1) estimated deaths in the US, and (2) estimated ICU bed requirements in the US.
I. IHME’s death projections are highly variable.
I focus on the March 26 and the April 2, 5, and 7 IHME projections of US COVID-19 deaths. The March 30-31 projections appear quite similar to March 26, and the April 1 update appears quite similar to April 2.
Here is a graph of projected daily deaths:
The first projection on March 26 (yellow) and the latest on April 7 (green) look quite similar, though they are not because the decline in the April 7 update is much faster. (This will be more evident in the next graph.) The April 2 update was a notable increase from March 26, and the April 5 update included a higher spike even than April 2, though it is also a shorter spike.
Here is the graph of projected cumulative deaths:
Here you can see the endpoint of each projection. The original March 26 projection (yellow) predicted 81,114 total deaths. This increased with the April 2 update (red), to 93,531. The April 5 update (blue) declined significantly (12.5%) to 81,766, though you can see the effect of the sharper spike in projected daily deaths in the more rapid increase in the blue line. The April 7 update (green) was a further significant decline (26.1%) to 60,415.
The total decline in projected deaths between the April 2 and the April 7 projection was over 35%. That’s a remarkably large change in a matter of five days.
The projections were quite wildly variable between states. A few examples:
- Massachusetts, which is currently projected to have the second-highest death total, changed from 2,382 in the April 2 update, to 8,254 in the April 5 update, to 5,625 in the April 7 update.
- New Jersey, which is currently projected to have the third-highest death total, changed from 2,117 in the April 2 update, to 9,691 in the April 5 update, to 5,277 in the April 7 update.
I’m not going to report the state subtotals, because they seem to be quite unreliable.
II. IHME’s ICU bed projections are highly variable and unreliable.
Another major feature of the IHME’s projections is the estimate of ICU bed requirements and shortfalls. They do a similar projection for overall hospital beds, but I’m going to focus on the ICU projections.
Here is the graph of the estimated number of ICU beds required for COVID-19 patients in the US:
You can see the large figure in the original March 26 projection (yellow), which increased in the April 2 update (red). Remember that the April 2 update was less than a week ago. The March 26 projection had a peak of 36,076 ICU beds on April 10, and the April 2 update increased this to a peak of 40,646 on April 12.
Both of the latest two updates have been significant declines. The April 5 update had a peak of 29,417 on April 16. Through this point, the projections had predicted a later date of peak ICU requirements. This changed with the April 7 update yesterday, which peaks at just 19,905 ICU beds on April 14.
Thus, projected ICU bed requirements dropped by more than 50% over the five days between the April 2 and the April 7 updates.
An even more important estimate presented by IHME is the projected shortage in ICU beds. This is the projection of the number of Americans that, according to IHME, would require ICU care but would be unable to receive such care. It is probably the most important single projection in the model, as a practical matter. Here is the graph::
You can see that the IHME predicted in its first release, on March 26 (yellow), that 17,380 Americans would be unable to obtain necessary ICU care on the peak date (which was then projected to be tomorrow, April 9). This estimate increased dramatically in the April 2 update (red) to 22,382 (to occur on April 10). The projection decreased and moved further into the future with the April 5 update (blue) to 16,433 (to occur on April 16).
The estimate decreased again with the April 7 update (green) to 9,047, predicted to occur on April 11. Thus, this prediction declined by about 60% over the five days between the April 2 and the April 7 updates.
It gets worse.
The IHME projections include a detailed projection for each individual state, day by day, which are totaled to create the nationwide figures cited above. I wondered whether the projected ICU shortage was heavily concentrated in certain states. It turns out that it is — New York and New Jersey. This should not be a surprise, because these two states have the highest number of reported deaths thus far.
The data allows me to determine the total number of available ICU beds in NY and NJ, according to the IHME projections. These figures are 718 for NY and 465 for NJ, a total of 1,183 available ICU beds in these two states combined.
I happen to have looked up information on the ICU shortage yesterday. According to this NYT article, New York state had 4,593 COVID-19 patients in the ICU yesterday (Tuesday, April 7), and New Jersey had 1,651. This was up slightly from 4,504 in NY and 1,505 in NJ on Monday, April 6.
The April 7 reported total for these two states, of COVID-19 patients actually in the ICU, was 6,244. Which is quite remarkable, since according to the IHME report released that same day, there would only be 1,183 ICU beds available to such patients.
Here is a graph of the projected need for ICU beds from the IHME updates on April 2 and April 7, showing the nationwide total and the total for just NY and NJ:
The red lines show the projected need for ICU beds in the April 2 update, for the entire US (top red line) and for just NY and NJ (lower red line). The green lines show the same projection in the April 7 update. The blue line is IHME’s estimate of total available ICU beds in NY and NJ. You can see that both of the lower lines (red and green) are far above the blue line. So much for “flattening the curve.”
Except, notice that highlighted yellow line (which actually connects two points). That is the actual number of COVID-19 patients in the ICU, in NY and NJ, on the past two days (April 6-7). Notice how it is vastly above the blue line, which is the number of available ICU beds assumed in the IHME model.
So I performed my own calculation on IHME’s projection of the shortage in ICU beds. They were obviously wrong about NY and NJ. What I did was to assume that, contrary to their model, there were 6,244 available ICU beds in NY and NJ for COVID-19 patients — not just 1,183 as assumed by IHME — which seems quite plausible in light of the NYT report that 6,244 COVID-19 patients were actually in the ICU in these two states yesterday.
Here’s how the ICU bed shortage graph looks, with my adjustment:
The red lines are the IHME’s projected shortage of ICU beds from its April 2 update, for the entire country and for just NY and NJ. The green lines are the same thing from the April 7 update. Note that in the April 7 update, most of the projected national shortage in ICU beds was in NY and NJ. Except, as noted above, there was no shortage in ICU beds in NY and NJ yesterday (April 7), according to the NYT.
The yellow lines are my calculations. What I did was to adjust IHME’s April 7 projection of the ICU bed shortage downward, as we know that at least 5,061 additional available ICU beds in NY and NJ (5,061 is the difference between ICU beds in NY and NJ actually occupied yesterday, 6,244, and the IHME’s assumption that only 1,183 ICU beds are available in these states).
You will notice that this one assumption virtually eliminates the projection for the ICU bed shortage. My top yellow line, the national projection, makes a “hump” because I did not adjust IHME’s assumptions for available ICU beds in other states (as I lack information to do so at this time). My calculation was not a blanket reduction of the national ICU shortage projection by 5,061 — rather, I reduced IHME’s April 7 projection (the top green line to my top yellow line) by the lesser of 5,061 or the number of ICU beds occupied in IHME’s projection for NY and NJ.
The lower yellow line — my adjusted estimate for NY and NJ — assumes that IHME’s projection of ICU beds needed in NY and NJ is accurate, and assumes that their ICU beds were fully occupied yesterday. I strongly suspect that this will not be the case.
My suspicion is that the actual shortage in ICU beds is zero, and should be zero in IHME’s projection. In my adjusted calculation, the peak ICU bed shortage in NY and NJ occurs tomorrow (April 9), at a shortage of 1,869 ICU beds. Of course, it is possible that the COVID-19 epidemic will exceed IHME’s latest projections, in which case it there might be a shortage of ICU beds.
Due to these deficiencies, I have concluded that IHME’s projection of ICU bed shortage is not only highly variable but is unreliable. It appears to be based upon a significant underestimate of total ICU bed availability.
I would like to be able to include a line in the graphs above for total nationwide ICU bed availability. This is not feasible from the information in the IHME model, as they do not report this figure directly. Rather, for some states, I can calculate their assumption regarding ICU bed availability as the difference between their projected ICU bed need and shortage. For many states, however, there is no shortage in the model, so I cannot calculate IHME’s assumed number of ICU beds in such states. Such a calculation would not be particularly helpful in any event, because ICU bed availability is a local issue — if you need an ICU bed in New Jersey, and there’s a bed available in Idaho, it doesn’t actually help.
III. Other issues with the IHME model
Two days ago, I posted my Deep Dive into the IHME model. I was particularly concerned that they appeared to have adjusted the model such that their “social distancing” measures no longer affected their estimated number of deaths, though it appeared that “social distancing” did affect the timing of such deaths. This occurred in a significant change in the IHME model released on April 5, and the update notes were ambiguous, so what they had done was unclear.
The April 5 update notes (here) stated that “more detailed information in the form of technical appendices will be published online by Tuesday, April 7,” and gave a link. It is now Wednesday afternoon, April 8, and the promised technical appendices are not available at the linked location, and do not appear to be available anywhere at the IHME site. This is disappointing.
It is also troubling that IHME does not appear to make its prior estimates available for download. It is possible that they are at the site, and I simply failed to find them. Transparency would dictate that they should “show their work,” including making an archive of their prior estimates readily accessible. (They do have such an archive for their update notes, which is good.) As noted at the outset, I had previously downloaded their original projection and all but one of the updates, so I had the information necessary to perform the analysis above.
IV. IMHE’s April 7 update included European totals
I have not analyzed them yet, but I noted that IHME’s April 7 update extended its projections to various European countries. Here is a quick summary for the larger nations of Western Europe.
Italy: IMHE states that Italy’s deaths may have peaked on March 27, and estimates a total of 20,300 in Italy (range 19,533-21,185). My data (from Johns Hopkins) showed Italy at 17,127 yesterday.
Spain: IHME states that Spain’s deaths may have peaked on April 1, and estimates a total of 19,209 in Spain (range 18,049-20,651). My data showed Spain at 14,045 yesterday.
France: IHME states that France’s deaths may have peaked on April 3, and estimates a total of 15,058 in France (range 12,906-17,715). My data showed France at 10,328 yesterday.
UK: IMHE projects the highest number of deaths, by a wide margin, in the UK. Their estimate is 66,314 (range 55,022-79,995). This is higher than their current estimate for the US, even without adjusting for population, and would be the proportional equivalent of about 325,000 deaths in the US. They project a peak in the UK on April 17. My data showed the UK at 6,159 yesterday.
Germany: IHME projects 8,802 deaths in Germany (range 4,253-21,350), with a peak on April 19. My data showed Germany at 2,016 yesterday.
I have not evaluated these estimates. It does seem very strange that IHME predicts such a high figure for the UK.
ChiCom delenda est.
Published in Healthcare
To be a bit of a contrarian on all this, there is still an argument that suggests COVID19 was already further spread at far earlier dates than generally used, and that would skew any models.
This came from a friend of a friend on Facebook, so at this point we’re talking 3 removes from me – take it with a grain of salt:
If true, this opens possibilities that COVID19 was already here in the US 5 months ago. If that’s the case, well….
I have hypothesized here in the past that what we’ve experienced in March could be a 2nd wave (akin to the 2nd, much more deadly, wave of the Spanish Flu); but I also recognize that the objections that Mendel articulated a few comments up would also apply to this hypothesis.
I think you need 80 percent exposure to achieve herd immunity
We can socially distance but not all of us need to stay home.
We need to test for immunity and antibodies.
Why can’t non essential businesses open at 50 percent capacity?
Because that would look like the politicians weren’t taking it seriously. Which is why we have garbage like blocking basketball hoops at city parks, lest someone come close to anyone else.
We need more tests like this. Your friend has recovered and is immune and is no longer a carrier.
He needs to be outside not locked inside
Too early to say. Take it one wave at a time
I’m a broken record on this, but whatever:
The initial IMHE model made no adjustment for the fact that spring was upon half the US at the onset of the outbreak, and subsequent models are frantically adjusting for the onset of spring in the other half.
It’s a theory at any rate, but I’ve yet to hear a better one.
new rules: 3 on 3, zone defense only, 3 point shots only
I think weather is definitely a factor that is being underrated.
Flu viruses in general do not like heat and humidity.
I believe the optimal temperature for most viruses is between 37 F and 63 F. Viruses also prefer being indoors vs outdoors, another reason why national self-quarantine must end soon.
I think Trump will announce on April 20 that ‘safe’ counties can re-open for business starting May 1.
LORD hear our prayer.
When I was a Senior in high school, a friend and I played a lot of basketball. One day we found a court that had 9 foot rims, so we were messing around, trying different dunks.
A bunch of 8th graders came up, four I think, and challenged us. We laughed. Then they said, “And you guys can’t dribble”. So we said, “That’s fine, but goaltending’s legal.”
We killed them (we were 6 inches taller than any of them). My buddy and I were throwing alley-oops to each other while they were shooting three point shots. That we would jump up through the rim and block.
That was a lot of fun.
and no one got the flu?
herd immunity while playing basketball
I know. It’s crazy.
The current numbers are still within the range of error reported back in March, so they are doing just fine with their projections, and they are updating them day by day as is appropriate.
The important thing to realize is that there is a lot of uncertainty, which they openly admit, and their mean projections may end up being too low as likely as too high.
I’m curious why this doesn’t make sense to you? There are lots of infectious diseases that are contagious before patients experience symptoms, or in which some people become contagious but never experience symptoms themselves. This isn’t a particularly new concept, although it’s certainly not the norm among respiratory viruses.
The more important question is: how important is asymptomatic transmission vs. symptomatic transmission in spreading the disease in real life? In other words, just because people can spread the disease without feeling any symptoms doesn’t mean this is the dominant form of transmission.
For example, if 1/3 of all new cases originated from a contagious person with no symptoms, whereas 2/3 originated from symptomatic patients, we could go a very long way towards “flattening the curve” by isolating symptomatic individuals despite knowingly leaving contagious people on the loose. Of course, that would require a psychological switch toward living with the virus instead of running away from it….
Granted, we don’t have enough solid data for that premise (as with so many premises). But it’s something worth keeping in mind.
I found this statement by Dr. Wittowski puzzling. Yes, respiratory viruses have been with us as long as we have been on this Earth. But some of those respiratory viruses have been incredibly deadly – not just the 1918 flu, but measles is actually a respiratory virus that’s hideously infectious and deadly if left unchecked. So I don’t think his point gets us anywhere. I do agree that the media is vastly overstating the danger of the virus, but that’s such a high bar that even taking the media’s hysteria down 50% still leaves us with a formidable public health threat.
As to “how deadly is it?”: a) nobody knows for sure because we need a lot more data than we currently have to provide a meaningful answer. All we can do is look at areas where it’s hit: in Spain and Italy, the death toll associated with the virus is truly many multiples of what we’re used to with seasonal respiratory viruses, and there’s enough corrobatory evidence to suggest that the virus is actually playing a causal role in the vast majority of those deaths, even if most victims were already old and/or unhealthy. On the other hand, we have Germany, which has a fairly high infection rate but a much lower death rate (albeit still likely several times higher than seasonal flu even taking undetected cases into account).
There are a million reasons to explain that discrepancy. In this case, viral sequence isn’t one of those: the majority of cases in Germany originate from the northern Italian strain.
But perhaps the take-home message should be: some diseases don’t have a single “mortality rate”, but rather, may be variably lethal depending on other factors. For example, untreated HIV infection has over a 90% fatality rate, but well-treated HIV infection has essentially 0% mortality. That’s why I prefer to think in terms of “what is Covid-19 capable of?”, not “how deadly is it really?”
Again, not enough data to really say. But here’s an important point I don’t think gets mentioned enough: most infectious diseases don’t spread homogeneously. Several of you have poo-poo’d the “superspreader” theory, but in essence every infectious disease is characterized by a small number of people who pass the germ onto quite a few others, and a large number of people who pass it on to many fewer. Again, HIV is a good example: when the disease first emerged, epidemiologists overestimated the general transmissibility based on the number of infected people they identified and the time range when they got infected. It turned out that there was a small number of, ahem, “superspreaders” who managed to pass on the disease to 25+ people, and lots of other infected people who didn’t pass it onto anybody despite unprotected sex.
The problem is that we usually end up averaging all of those into a single “R0” figure that really doesn’t encapsulate the complexity of biology but exudes an illusion of homogeneous spread.
What we do know is that there have been several well-documented examples in which a single infected person has managed to lead to many hundreds or even thousands of infections within a very short amount of time. The bar in Austria is one example, the Christian group in South Korea is another well-documented example.
In a way, super-spreading may actually be good news, because it means everyday transmission (like at the grocery store or on a little league team) is actually much less dangerous than it initially appeared. This in turn would mean that we could reduce transmission greatly by surgically banning certain types of interactions/events while leaving other normal modes of interaction relatively untouched.
Bottom line: we should actually be rooting for superspreading to be a major cause of transmission.
Fair enough.
The basic issue is that in order to apply scientific method, we would need much more data (and reliable data) than is currently available. Until that data becomes available, it may be too late to save the economy. That’s why my plea is for us to adopt the criteria of “which risks are plausible enough that we should take them into serious account when deciding how to rapidly reopen society”. And in my opinion, that type of risk assessment would need to adopt the working hypothesis that social distancing works based on the data available – even if that hypothesis turns out later to be false (or at least less positive than imagined).
Also, just in case we needed more evidence of how useless the IHME model has proven to be:
I hope it’s clear that while I’m skeptical about many of the hypotheses suggesting Covid-19 is only marginally more harmful than seasonal flu, I’m outright disdainful of these projections – especially the fact that their authors have not been vociferously reiterating how uncertain they are. In my opinion, that is absolute malpractice.