Technological
innovation sometimes requires we take risks — and question the tenets of
evidence-based medicine.
John
Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior
research analyst and communications specialist, Mayo Clinic Platform, wrote
this article.
It’s
challenging at times to know when to follow the rules and “color inside the
lines” and when to ignore those lines and forge ahead. That’s true whether
we’re navigating everyday life, creating new technology, or devising the best
patient care initiatives. Which brings to mind a quote from Elbert Hubbard: “The greatest mistake you can make in life is continually fearing that
you'll make one.”
Over
the years, we have discussed the strengths and weaknesses of evidence-based
medicine and randomized controlled trials (RCTs) in several books and articles,
our point being that fear of investing in a treatment approach because it not
supported by the RCT “gold standard” can create a kind of inertia that
ultimately hurts patients.1,2 And even if we put aside the fear factor that
Hubbard mentions, mounting evidence strongly suggests that an in-depth analysis
of large data sets can supplement — and in some cases be substituted for — RCTs to
support the clinical decision-making process.
That
doesn’t imply that RCTs should be abandoned.
The list of treatments that have been supported or retired due to a
well-designed RCT is long. For decades, surgeons used radical mastectomy to
treat breast cancer until a controlled trial demonstrated that less disfiguring
alternatives were just as effective in managing the disease.3 Similarly, clinicians used to freely prescribe hormone
replacement therapy to women in menopause until the Women’s Health Initiative, also
an RCT, demonstrated that it increases the risk of heart disease, stroke, and
breast cancer. But on the other hand, there have been many recent non-RCT investigations
that have taken advantage of the power of massive data sets and have generated
actionable insights. With the VA Boston Health System, Julia Prentice and her
colleagues, using administrative data from more than 80,000 veterans, have
shown that among patients with Type 2 diabetes, sulfonylurea drugs increased
the risk of dying or being hospitalized when compared to patients on thiazolidinediones.4 Similarly, David Graham created a stir when he analyzed
the patient records of approximately 1.4 million patients who belonged to
Kaiser Permanente in California. They aimed to determine if rofecoxib (Vioxx)
increased the risk of acute myocardial infarction and sudden cardiac death.
Graham et al. reviewed the equivalent of 2,302,029 person-years of follow-up. They detected 8,142 cases of serious
coronary heart disease (CHD), 2,210 of which were fatal. The odds of developing
CHD among patients taking any dose of the medication were 59% greater than it
was among controls. Among patients who took high doses, namely more than 25 mg
daily, the odds of heart disease were 258% greater. 5
More recently, nference, a data
analytics firm with a partnership with Mayo Clinic, spearheaded several
data-intensive studies that did not use the traditional clinical trials
protocol. One study used deep neural networks to evaluate 15.8 million clinical
notes in an EHR from over 30,000 patients who underwent COVID-19 diagnostic
testing.6 When investigators compared patients with clinically
apparent COVID-19 with negative patients about a week before they had PCR
testing to confirm the diagnoses, they found loss of taste and smell was more
than 37-fold more likely to occur in those whose infection was confirmed versus
those who tested negative. Shweta et al. state, “This
study introduces an augmented intelligence platform for the real-time synthesis
of institutional knowledge captured in EHRs.” One caveat that the researchers acknowledge
in the report was that they had yet to conduct prospective validation of the
augmented EHR curation approach.
A
second nference-based investigation reviewed the records of patients who had
received more than 94,000 doses of the Pfizer COVID-19 vaccine, more than 36,000
doses of the Moderna vaccine, and 1,745 doses of the Johnson & Johnson
vaccine. The study’s goal was to determine the incidence of cerebral venous
sinus thrombosis (CVST), which has been reported in a small number of patients
after receiving one of the vaccines.7 The preprint study
found no significant association between any of the vaccines as CVST.
One
of the strengths of RCTs is their prospective nature, a design that is more
likely to eliminate confounding variables and bias when compared to
retrospective studies. But at the same time, several RCTs have fallen short
because they were underpowered, resulting in false-negative results. Also, RCTs
are expensive and often require many years to generate results that clinicians
can use at the bedside. On the other hand, retrospective analyses can generate
results much more quickly, and under the right circumstances, can provide
actionable insights and inform the clinical decision-making process.
Thomas
Frieden, MD, MPH, a former director of the CDC, has pointed out the real-world
advantages of retrospective cohort studies, which have been used to assess the
prognosis and treatment of various types of cancer. That, in turn, has led to
better treatment protocols. Similarly, such cohort studies have successfully
been used to evaluate survival among pediatric cancer patients and made
clinicians aware of the “increased risk of post-treatment cardiac
complications, enabling better clinical care.”8 Frieden summed up the controversy this way, “Elevating
RCTs at the expense of other potentially highly valuable sources of data is
counterproductive. A better approach is to clarify the health outcome being
sought and determine whether existing data are available that can be rigorously
and objectively evaluated, independently of or in comparison with data from
RCTs, or whether new studies (RCT or otherwise) are needed.”
When comparing research methodologies,
it’s important to remember that’s it’s not a sports competition; there doesn’t
have to be a clear winner and loser. Big data analytics and RCTs each have
their strengths and weaknesses and can be deployed accordingly. When there's
enough time and resources available to conduct a controlled trial, it is often
the best way to evaluate potentially useful treatment approaches. Still, when
clinicians need to quickly make diagnostic and therapeutic decisions,
especially during an international crisis, we don't always have the luxury of
time.
References
1. Cerrato P,
Halamka J. Realizing the Promise of Precision Medicine. 2017, Academic
Press/Elsevier, Cambridge, MA, pp. 87-91.
2. Cerrato,
P, Halamka J. The Transformative Power of Mobile Medicine. 2019,
Academic Press/Elsevier, Cambridge, MA, pp 57-58.
3. Treasure
T, Takkenberg JM. Randomized trials and big data analysis: we need the best of both worlds. Eur J
CardioThoracic Surg. 2018; 53:910-914.
4. Prentice
JC, Conlin PR, Gellad WF et al. Capitalizing on Prescribing Pattern Variation
to Compare Medications forType2Diabetes. Value in Health. 2014;
17:854-862.
5. Graham DJ,
Campen D, Hui R, et al. Risk of acute myocardial infarction and sudden cardiac
death in patients treated with cyclo-oxygenase 2 selective and nonselective
non-steroidal anti-inflammatory drugs: nested case-control study. Lancet
2005;365:475–581.
6. Shweta F,
Murugadoss K, Awasthi S el al. Augmented Curation of Unstructured Clinical
Notes from a Massive EHR System Reveals Specific Phenotypic Signature of
Impending COVID-19 Diagnosis. eLife. Published online July 7, 2020. https://elifesciences.org/articles/58227
7. Pawlowski
C, Rincon-Hekling J, et al. Cerebral venous sinus thrombosis (CVST) is not
significantly linked to COVID-19 vaccines or non-COVID vaccines in a large
multi-state US health system. medRxiv. 2021, April 23. https://www.medrxiv.org/content/10.1101/2021.04.20.21255806v1
8. Frieden
TR. Evidence for Health Decision Making —Beyond Randomized, Controlled Trials. N
Engl. J Med.2017;377:465-475.