Ricochet is the best place on the internet to discuss the issues of the day, either through commenting on posts or writing your own for our active and dynamic community in a fully moderated environment. In addition, the Ricochet Audio Network offers over 50 original podcasts with new episodes released every day.
As most of you know, the Institute for Health Metrics and Evaluation (IHME) at the University of Washington has published a series of COVID-19 projections over the past couple of weeks. The IHME projections are sometimes called the “Chris Murray” model, after its director. It appears that these projections are being widely used by the Trump administration and other government officials. The main site is here.
The IHME released its original projections on March 26, and has released updates on March 30 and 31 and April 1, 2, 5, and 7. It appears that only the most recent projection is available for download at the site. From previous downloads, I have all of them except April 1 (which appears quite similar to April 2, based on the update notes that are available for each update here).
This post is an evaluation of the changes in the IHME projections. I focus on two issues: (1) estimated deaths in the US, and (2) estimated ICU bed requirements in the US.
I. IHME’s death projections are highly variable.
I focus on the March 26 and the April 2, 5, and 7 IHME projections of US COVID-19 deaths. The March 30-31 projections appear quite similar to March 26, and the April 1 update appears quite similar to April 2.
Here is a graph of projected daily deaths:
The first projection on March 26 (yellow) and the latest on April 7 (green) look quite similar, though they are not because the decline in the April 7 update is much faster. (This will be more evident in the next graph.) The April 2 update was a notable increase from March 26, and the April 5 update included a higher spike even than April 2, though it is also a shorter spike.
Here is the graph of projected cumulative deaths:
Here you can see the endpoint of each projection. The original March 26 projection (yellow) predicted 81,114 total deaths. This increased with the April 2 update (red), to 93,531. The April 5 update (blue) declined significantly (12.5%) to 81,766, though you can see the effect of the sharper spike in projected daily deaths in the more rapid increase in the blue line. The April 7 update (green) was a further significant decline (26.1%) to 60,415.
The total decline in projected deaths between the April 2 and the April 7 projection was over 35%. That’s a remarkably large change in a matter of five days.
The projections were quite wildly variable between states. A few examples:
- Massachusetts, which is currently projected to have the second-highest death total, changed from 2,382 in the April 2 update, to 8,254 in the April 5 update, to 5,625 in the April 7 update.
- New Jersey, which is currently projected to have the third-highest death total, changed from 2,117 in the April 2 update, to 9,691 in the April 5 update, to 5,277 in the April 7 update.
I’m not going to report the state subtotals, because they seem to be quite unreliable.
II. IHME’s ICU bed projections are highly variable and unreliable.
Another major feature of the IHME’s projections is the estimate of ICU bed requirements and shortfalls. They do a similar projection for overall hospital beds, but I’m going to focus on the ICU projections.
Here is the graph of the estimated number of ICU beds required for COVID-19 patients in the US:
You can see the large figure in the original March 26 projection (yellow), which increased in the April 2 update (red). Remember that the April 2 update was less than a week ago. The March 26 projection had a peak of 36,076 ICU beds on April 10, and the April 2 update increased this to a peak of 40,646 on April 12.
Both of the latest two updates have been significant declines. The April 5 update had a peak of 29,417 on April 16. Through this point, the projections had predicted a later date of peak ICU requirements. This changed with the April 7 update yesterday, which peaks at just 19,905 ICU beds on April 14.
Thus, projected ICU bed requirements dropped by more than 50% over the five days between the April 2 and the April 7 updates.
An even more important estimate presented by IHME is the projected shortage in ICU beds. This is the projection of the number of Americans that, according to IHME, would require ICU care but would be unable to receive such care. It is probably the most important single projection in the model, as a practical matter. Here is the graph::
You can see that the IHME predicted in its first release, on March 26 (yellow), that 17,380 Americans would be unable to obtain necessary ICU care on the peak date (which was then projected to be tomorrow, April 9). This estimate increased dramatically in the April 2 update (red) to 22,382 (to occur on April 10). The projection decreased and moved further into the future with the April 5 update (blue) to 16,433 (to occur on April 16).
The estimate decreased again with the April 7 update (green) to 9,047, predicted to occur on April 11. Thus, this prediction declined by about 60% over the five days between the April 2 and the April 7 updates.
It gets worse.
The IHME projections include a detailed projection for each individual state, day by day, which are totaled to create the nationwide figures cited above. I wondered whether the projected ICU shortage was heavily concentrated in certain states. It turns out that it is — New York and New Jersey. This should not be a surprise, because these two states have the highest number of reported deaths thus far.
The data allows me to determine the total number of available ICU beds in NY and NJ, according to the IHME projections. These figures are 718 for NY and 465 for NJ, a total of 1,183 available ICU beds in these two states combined.
I happen to have looked up information on the ICU shortage yesterday. According to this NYT article, New York state had 4,593 COVID-19 patients in the ICU yesterday (Tuesday, April 7), and New Jersey had 1,651. This was up slightly from 4,504 in NY and 1,505 in NJ on Monday, April 6.
The April 7 reported total for these two states, of COVID-19 patients actually in the ICU, was 6,244. Which is quite remarkable, since according to the IHME report released that same day, there would only be 1,183 ICU beds available to such patients.
Here is a graph of the projected need for ICU beds from the IHME updates on April 2 and April 7, showing the nationwide total and the total for just NY and NJ:
The red lines show the projected need for ICU beds in the April 2 update, for the entire US (top red line) and for just NY and NJ (lower red line). The green lines show the same projection in the April 7 update. The blue line is IHME’s estimate of total available ICU beds in NY and NJ. You can see that both of the lower lines (red and green) are far above the blue line. So much for “flattening the curve.”
Except, notice that highlighted yellow line (which actually connects two points). That is the actual number of COVID-19 patients in the ICU, in NY and NJ, on the past two days (April 6-7). Notice how it is vastly above the blue line, which is the number of available ICU beds assumed in the IHME model.
So I performed my own calculation on IHME’s projection of the shortage in ICU beds. They were obviously wrong about NY and NJ. What I did was to assume that, contrary to their model, there were 6,244 available ICU beds in NY and NJ for COVID-19 patients — not just 1,183 as assumed by IHME — which seems quite plausible in light of the NYT report that 6,244 COVID-19 patients were actually in the ICU in these two states yesterday.
Here’s how the ICU bed shortage graph looks, with my adjustment:
The red lines are the IHME’s projected shortage of ICU beds from its April 2 update, for the entire country and for just NY and NJ. The green lines are the same thing from the April 7 update. Note that in the April 7 update, most of the projected national shortage in ICU beds was in NY and NJ. Except, as noted above, there was no shortage in ICU beds in NY and NJ yesterday (April 7), according to the NYT.
The yellow lines are my calculations. What I did was to adjust IHME’s April 7 projection of the ICU bed shortage downward, as we know that at least 5,061 additional available ICU beds in NY and NJ (5,061 is the difference between ICU beds in NY and NJ actually occupied yesterday, 6,244, and the IHME’s assumption that only 1,183 ICU beds are available in these states).
You will notice that this one assumption virtually eliminates the projection for the ICU bed shortage. My top yellow line, the national projection, makes a “hump” because I did not adjust IHME’s assumptions for available ICU beds in other states (as I lack information to do so at this time). My calculation was not a blanket reduction of the national ICU shortage projection by 5,061 — rather, I reduced IHME’s April 7 projection (the top green line to my top yellow line) by the lesser of 5,061 or the number of ICU beds occupied in IHME’s projection for NY and NJ.
The lower yellow line — my adjusted estimate for NY and NJ — assumes that IHME’s projection of ICU beds needed in NY and NJ is accurate, and assumes that their ICU beds were fully occupied yesterday. I strongly suspect that this will not be the case.
My suspicion is that the actual shortage in ICU beds is zero, and should be zero in IHME’s projection. In my adjusted calculation, the peak ICU bed shortage in NY and NJ occurs tomorrow (April 9), at a shortage of 1,869 ICU beds. Of course, it is possible that the COVID-19 epidemic will exceed IHME’s latest projections, in which case it there might be a shortage of ICU beds.
Due to these deficiencies, I have concluded that IHME’s projection of ICU bed shortage is not only highly variable but is unreliable. It appears to be based upon a significant underestimate of total ICU bed availability.
I would like to be able to include a line in the graphs above for total nationwide ICU bed availability. This is not feasible from the information in the IHME model, as they do not report this figure directly. Rather, for some states, I can calculate their assumption regarding ICU bed availability as the difference between their projected ICU bed need and shortage. For many states, however, there is no shortage in the model, so I cannot calculate IHME’s assumed number of ICU beds in such states. Such a calculation would not be particularly helpful in any event, because ICU bed availability is a local issue — if you need an ICU bed in New Jersey, and there’s a bed available in Idaho, it doesn’t actually help.
III. Other issues with the IHME model
Two days ago, I posted my Deep Dive into the IHME model. I was particularly concerned that they appeared to have adjusted the model such that their “social distancing” measures no longer affected their estimated number of deaths, though it appeared that “social distancing” did affect the timing of such deaths. This occurred in a significant change in the IHME model released on April 5, and the update notes were ambiguous, so what they had done was unclear.
The April 5 update notes (here) stated that “more detailed information in the form of technical appendices will be published online by Tuesday, April 7,” and gave a link. It is now Wednesday afternoon, April 8, and the promised technical appendices are not available at the linked location, and do not appear to be available anywhere at the IHME site. This is disappointing.
It is also troubling that IHME does not appear to make its prior estimates available for download. It is possible that they are at the site, and I simply failed to find them. Transparency would dictate that they should “show their work,” including making an archive of their prior estimates readily accessible. (They do have such an archive for their update notes, which is good.) As noted at the outset, I had previously downloaded their original projection and all but one of the updates, so I had the information necessary to perform the analysis above.
IV. IMHE’s April 7 update included European totals
I have not analyzed them yet, but I noted that IHME’s April 7 update extended its projections to various European countries. Here is a quick summary for the larger nations of Western Europe.
Italy: IMHE states that Italy’s deaths may have peaked on March 27, and estimates a total of 20,300 in Italy (range 19,533-21,185). My data (from Johns Hopkins) showed Italy at 17,127 yesterday.
Spain: IHME states that Spain’s deaths may have peaked on April 1, and estimates a total of 19,209 in Spain (range 18,049-20,651). My data showed Spain at 14,045 yesterday.
France: IHME states that France’s deaths may have peaked on April 3, and estimates a total of 15,058 in France (range 12,906-17,715). My data showed France at 10,328 yesterday.
UK: IMHE projects the highest number of deaths, by a wide margin, in the UK. Their estimate is 66,314 (range 55,022-79,995). This is higher than their current estimate for the US, even without adjusting for population, and would be the proportional equivalent of about 325,000 deaths in the US. They project a peak in the UK on April 17. My data showed the UK at 6,159 yesterday.
Germany: IHME projects 8,802 deaths in Germany (range 4,253-21,350), with a peak on April 19. My data showed Germany at 2,016 yesterday.
I have not evaluated these estimates. It does seem very strange that IHME predicts such a high figure for the UK.
ChiCom delenda est.Published in