It’s OK to Break the Rules Now and Then

Technological innovation sometimes requires we take risks — and question the tenets of evidence-based medicine.

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

It’s challenging at times to know when to follow the rules and “color inside the lines” and when to ignore those lines and forge ahead. That’s true whether we’re navigating everyday life, creating new technology, or devising the best patient care initiatives. Which brings to mind a quote from Elbert Hubbard: “The greatest mistake you can make in life is continually fearing that you'll make one.” 

Over the years, we have discussed the strengths and weaknesses of evidence-based medicine and randomized controlled trials (RCTs) in several books and articles, our point being that fear of investing in a treatment approach because it not supported by the RCT “gold standard” can create a kind of inertia that ultimately hurts patients.^1,2  And even if we put aside the fear factor that Hubbard mentions, mounting evidence strongly suggests that an in-depth analysis of large data sets can supplement — and in some cases be substituted for — RCTs to support the clinical decision-making process.

That doesn’t imply that RCTs should be abandoned.  The list of treatments that have been supported or retired due to a well-designed RCT is long. For decades, surgeons used radical mastectomy to treat breast cancer until a controlled trial demonstrated that less disfiguring alternatives were just as effective in managing the disease.³ Similarly, clinicians used to freely prescribe hormone replacement therapy to women in menopause until the Women’s Health Initiative, also an RCT, demonstrated that it increases the risk of heart disease, stroke, and breast cancer. But on the other hand, there have been many recent non-RCT investigations that have taken advantage of the power of massive data sets and have generated actionable insights. With the VA Boston Health System, Julia Prentice and her colleagues, using administrative data from more than 80,000 veterans, have shown that among patients with Type 2 diabetes, sulfonylurea drugs increased the risk of dying or being hospitalized when compared to patients on thiazolidinediones.⁴ Similarly, David Graham created a stir when he analyzed the patient records of approximately 1.4 million patients who belonged to Kaiser Permanente in California. They aimed to determine if rofecoxib (Vioxx) increased the risk of acute myocardial infarction and sudden cardiac death. Graham et al. reviewed the equivalent of 2,302,029 person-years of follow-up. They detected 8,142 cases of serious coronary heart disease (CHD), 2,210 of which were fatal. The odds of developing CHD among patients taking any dose of the medication were 59% greater than it was among controls. Among patients who took high doses, namely more than 25 mg daily, the odds of heart disease were 258% greater. ⁵

More recently, nference, a data analytics firm with a partnership with Mayo Clinic, spearheaded several data-intensive studies that did not use the traditional clinical trials protocol. One study used deep neural networks to evaluate 15.8 million clinical notes in an EHR from over 30,000 patients who underwent COVID-19 diagnostic testing.⁶ When investigators compared patients with clinically apparent COVID-19 with negative patients about a week before they had PCR testing to confirm the diagnoses, they found loss of taste and smell was more than 37-fold more likely to occur in those whose infection was confirmed versus those who tested negative. Shweta et al. state, “This study introduces an augmented intelligence platform for the real-time synthesis of institutional knowledge captured in EHRs.” One caveat that the researchers acknowledge in the report was that they had yet to conduct prospective validation of the augmented EHR curation approach.

A second nference-based investigation reviewed the records of patients who had received more than 94,000 doses of the Pfizer COVID-19 vaccine, more than 36,000 doses of the Moderna vaccine, and 1,745 doses of the Johnson & Johnson vaccine. The study’s goal was to determine the incidence of cerebral venous sinus thrombosis (CVST), which has been reported in a small number of patients after receiving one of the vaccines.⁷ The preprint study found no significant association between any of the vaccines as CVST.

One of the strengths of RCTs is their prospective nature, a design that is more likely to eliminate confounding variables and bias when compared to retrospective studies. But at the same time, several RCTs have fallen short because they were underpowered, resulting in false-negative results. Also, RCTs are expensive and often require many years to generate results that clinicians can use at the bedside. On the other hand, retrospective analyses can generate results much more quickly, and under the right circumstances, can provide actionable insights and inform the clinical decision-making process.

Thomas Frieden, MD, MPH, a former director of the CDC, has pointed out the real-world advantages of retrospective cohort studies, which have been used to assess the prognosis and treatment of various types of cancer. That, in turn, has led to better treatment protocols. Similarly, such cohort studies have successfully been used to evaluate survival among pediatric cancer patients and made clinicians aware of the “increased risk of post-treatment cardiac complications, enabling better clinical care.”⁸ Frieden summed up the controversy this way, “Elevating RCTs at the expense of other potentially highly valuable sources of data is counterproductive. A better approach is to clarify the health outcome being sought and determine whether existing data are available that can be rigorously and objectively evaluated, independently of or in comparison with data from RCTs, or whether new studies (RCT or otherwise) are needed.”

When comparing research methodologies, it’s important to remember that’s it’s not a sports competition; there doesn’t have to be a clear winner and loser. Big data analytics and RCTs each have their strengths and weaknesses and can be deployed accordingly. When there's enough time and resources available to conduct a controlled trial, it is often the best way to evaluate potentially useful treatment approaches. Still, when clinicians need to quickly make diagnostic and therapeutic decisions, especially during an international crisis, we don't always have the luxury of time.

 

References

1. Cerrato P, Halamka J. Realizing the Promise of Precision Medicine. 2017, Academic Press/Elsevier, Cambridge, MA, pp. 87-91.

2. Cerrato, P, Halamka J. The Transformative Power of Mobile Medicine. 2019, Academic Press/Elsevier, Cambridge, MA, pp 57-58.

3. Treasure T, Takkenberg JM. Randomized trials and big data analysis: we need the best of both worlds. Eur J CardioThoracic Surg. 2018; 53:910-914.

4. Prentice JC, Conlin PR, Gellad WF et al. Capitalizing on Prescribing Pattern Variation to Compare Medications forType2Diabetes. Value in Health. 2014; 17:854-862.

5. Graham DJ, Campen D, Hui R, et al. Risk of acute myocardial infarction and sudden cardiac death in patients treated with cyclo-oxygenase 2 selective and nonselective non-steroidal anti-inflammatory drugs: nested case-control study. Lancet 2005;365:475–581.

6. Shweta F, Murugadoss K, Awasthi S el al. Augmented Curation of Unstructured Clinical Notes from a Massive EHR System Reveals Specific Phenotypic Signature of Impending COVID-19 Diagnosis. eLife. Published online July 7, 2020. https://elifesciences.org/articles/58227

7. Pawlowski C, Rincon-Hekling J, et al. Cerebral venous sinus thrombosis (CVST) is not significantly linked to COVID-19 vaccines or non-COVID vaccines in a large multi-state US health system. medRxiv. 2021, April 23. https://www.medrxiv.org/content/10.1101/2021.04.20.21255806v1

8. Frieden TR. Evidence for Health Decision Making —Beyond Randomized, Controlled Trials. N Engl. J Med.2017;377:465-475.