Get the Ricochet app
Ricochet will work fine in a browser on your mobile devices, but for the best experience please download our app.
There’s an old joke among neurosurgeons: if you present a case to five surgeons, you’ll get seven different opinions.
Consider a patient with neck pain and a herniated disc. The clinical nuance is immense. There are nonoperative options: physical therapy, medications, steroid injections. There are surgical options ranging from limited decompressions to artificial disc replacement to multilevel fusion with rods and screws. Each choice involves tradeoffs that depend on anatomy, symptoms, risk tolerance, lifestyle, and values.
Then there is execution. Was the surgery performed efficiently and precisely, or slowly and sloppily? A patient may undergo a fusion performed flawlessly from a technical standpoint and still endure a prolonged recovery or postoperative infection. Another patient may choose nonoperative care and continue to suffer, or experience spontaneous improvement. Outcomes vary even when decisions are reasonable.
Which raises the question patients and policymakers keep asking: Who was the better doctor?
For decades, an entire quality metric industrial complex has tried to answer this question. It relies on what is easiest to measure: readmission rates, length of stay, and complication codes. These metrics are “risk-adjusted” using administrative data, but only by adjusting for variables that themselves are easily measured.
We are obsessed with finding a “ground truth” for physician quality. But how would one actually recognize a good doctor? Ask patients, and the answer is inherently subjective. Different patients want different things at different times. Most want evidence-based recommendations, but many do not. (As proof, the healing crystal industry in the U.S. is worth over $1 billion annually.)
Ask bureaucrats, and “good” is defined by costs, compliance, and checkboxes.
Our current system relies on absolute judgment. We take a single physician and grade them against a fixed rubric devised by private contractors working for governments and insurers. Unsurprisingly, this results in measuring what is easy to measure rather than what matters. Clinical judgment, technical finesse, and empathy remain largely invisible.
It also turns out humans are terrible at absolute judgment.
This is well illustrated in an EconTalk discussion featuring Daisy Christodoulou. She uses soccer as an analogy. Was a particular tackle “too rough” and deserving of a yellow card? Even with slow-motion replay, experts frequently disagree when evaluating a single event in isolation. A better approach is comparative judgment. Instead of asking whether a tackle crosses some abstract threshold, ask a simpler question: Which of these two tackles was rougher? Humans are remarkably good at that, and the same principle can apply to medicine.
Rather than grading a surgeon’s care of cervical spine disease as an “A–” or “B+,” we can compare two cases and ask: Who managed this case better? Repeat this process thousands of times, and a statistical pattern emerges. This approach respects the inherent nuance of medicine. It allows for stylistic differences while still punishing objectively bad care, which reliably appears as an outlier that loses nearly every comparison.
The deeper problem is not that we have chosen the wrong metrics. It is that absolute physician quality does not exist as a stable quantity to be measured.
Absolute judgment requires a fixed standard: the same patient preferences, the same constraints, the same tradeoffs, and the same definition of success. Medicine offers none of these. Outcomes depend on anatomy, comorbidities, risk tolerance, timing, social support, and values, many of which are unobservable and irreducible to data.
Without a fixed reference frame, there is no such thing as an “A-level doctor” in the abstract. There are only doctors performing better or worse relative to other doctors facing similar problems. Any attempt to assign an absolute score is therefore not merely imprecise but conceptually incoherent.
This is not intuitive. It rests on nearly a century of statistical theory.
In 1927, psychologist Louis Thurstone proposed the Law of Comparative Judgment. He observed that human evaluation of a single object is noisy and unstable, influenced by mood, context, and bias. But when forced to choose between two objects, those errors tend to cancel out, producing far more reliable judgments.
These binary comparisons can then be analyzed using the Bradley–Terry model, which takes win–loss data and infers a latent “strength” or “quality” parameter for each participant. This is the same mathematics underlying Elo ratings in chess and modern matchmaking systems. We do not need a checklist to know Magnus Carlsen is good at chess. We know because he consistently beats nearly everyone he faces. The math simply formalizes that reality.
Applied to medicine, this framework would free us from reaching to assess “quality” in the abstract. Quality would be the latent variable that best explains why one physician’s decisions and outcomes are consistently preferred over another’s by their peers.
This approach is not easy. Comparative judgment does not scale cleanly and cannot be administered by bureaucrats. That is precisely why the establishment will resist it. Perhaps one day AI will assist in this process, but that day has not yet arrived. Good medicine is not a checklist. Quality is not a variable sitting in the electronic health record waiting to be mined. It is a consensus.
We cannot measure doctors against a divine standard. We can only measure them against each other. And that may be the closest thing to ground truth we will ever get.
Discussion (17)
To read this bio, please join Ricochet.
The second mainframe COBOL software development project I was assigned to in 1979 after moving to America the previous year was at a Park Avenue consulting company which had a contract with New York state to analyze some of its healthcare expenditures. We were given patient admission, discharge, and billing data for a calendar year for 41 hospitals of varying types—urban, rural, union, non-union, profit, non-profit, and so on—and the DRG code or codes associated with each patient throughout their stay (diagnosis-related groups is a taxonomic system that assigns a unique code to hospital cases—diseases, conditions, etc.).
We didn’t have a stats package, so to calculate means, standard deviations, etc., we had to write one ourselves. All spreadsheet and non-spreadsheet output (histograms, distribution curves, and the like) was printed on greenbar paper, the latter using characters such as hyphens, asterisks, and equal signs as graphic output wasn’t yet a thing. The data presented its own problems: discharge records but no admission records for people already in hospital on Jan. 1; admission records but no discharge records for people still in hospital on Dec. 31; admission records but no discharge records for people who died under care; a handful of cases for which we had neither an admission record nor a discharge record because the patient entered care before Jan. 1 and was still in hospital on Dec. 31. It would have been easy to simply exclude these anomalies, but we had cost records for these patients and couldn’t ignore the money the state had paid for treatment, drugs, and the like, so we invented our own DRGs for them.
One result of the study that struck me at the time concerned hospital births. Because American medicine has vaginal delivery in a hospital down pat, the length-of-stay distribution curve for the DRG normal live birth had a pronounced central hump with narrow shoulders and almost no outliers: mothers were almost guaranteed to be in hospital for three or four days or whatever it was, and no more. If you were admitted for cerebral hemorrhage, however, you might be there for a few days or several weeks. Easy to predict, perhaps, but I found it fascinating on my first medical computing gig to explore these differences.
Anyone on the hook for medical expenditures—patients, insurance companies, states, and the federal government—should be able to look at cost and length-of-stay numbers—perhaps adjusted for case mix—for both hospitals and physicians, especially surgeons, and especially for routine operations like tonsillectomies, simple fractures, cataract removals, vasectomies, and the like. An informed consumer, etc.
Note: This comment follows the British convention of saying “in hospital” rather than the silly American usage, “in the hospital.” Americans are quite happy to describe someone as being “in college” but insist on adding the definite article to “hospital.” It’s doubly silly because if they’re going to add unnecessary words they should use the indefinite article and say “in a hospital,” as in most situations the hospital concerned hasn’t been identified by name or location.
To read this bio, please join Ricochet.
Almost nothing has a universal "good." Everything has tradeoffs. And those tradeoffs depend a lot on individual or personal circumstances, personal preferences, personal goals, geographic factors, personal or group risk tolerance, and numerous other factors.
I am amazed at the frequency with which questions appear in internet advice sites that are based on assumptions that whatever the topic there is always a "best" answer. What is the "best" car? Is it "better" to do financial task A versus financial task B? Or to take professional path A versus professional path B?
This insistence that there is a "best" practice for everyone is a central fallacy of the Progressive program of government by technocrats.
What is a "best practice" for a person living in population dense Manhattan (New York City) or Washington, D.C., is likely to be very different from a "best practice" for either of my children, both of whom live in small towns in New Mexico.
Even between my two children, who both live in small towns a mere one hundred miles apart, one has a strong interest in living near the center of town so she and her husband can walk or bicycle to town social events. The other has a strong interest in having space in which he and his wife can have livestock on premises, and have little interest in socializing with other humans in town.
So, of course there are going to be a variety of measures for what is "best" or even "good" medical care.
To read this bio, please join Ricochet.
Once you give insurers the right of refusal you distort the data. If you are required to pursue physical therapy or oral analgesics for a specified time before you can do an injection or a procedure you introduce variables that over ride physician judgement and patient interest and need. Insurers seem to assume that physicians are going to advise the care that nets them the greatest income while physicians see the insurer as seeking to minimize payouts strictly to their financial advantages. Physician skill and judgment and patient need and preferences are quashed. There is no possibility of assessing quality in such a system. Nor if delivering better care.
I am told by my expert peers that I should screen for and treat metabolic associated streatotic liver disease but insurers will not cover such (expensive) treatment except in narrow circumstances. And that is for a disease that affects 30% of the population. Not possible to deliver appropriate care in such a system.
CONTRIBUTOR
To read this bio, please join Ricochet.
That fallacy is present to a greater or lesser extent in every last cell of our body economic.
To read this bio, please join Ricochet.
A similar subject came up in a Ricochet thread many years ago. I pointed out that it is really easy to determine who is the fastest runner in a race, or the strongest lifter in a weightlifting event, or the fastest solver of math problems. But how do you determine the best Mother? Or the best Doctor?
I had a couple of smart-alek Ricochet doctors answer my second question with "Well of course, I am!"
You're seeing five of 17 comments. Sign in, or join for just $5 per month, to read them all and add your own.