Tuesday, March 2, 2021

Listen Better, See Deeper

Combining Medical Attentiveness with Artificial Intelligence

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

Embracing an “ecology of attention” will significantly improve patient care, according to Mark Kissler, MD, at the University of Colorado.1 Kissler and his colleagues point out that clinicians spend much of their time multi-tasking and navigating around interruptions. While such juggling acts are often unavailable, it’s important to occasionally step back and ask: Is this the best use of my time? Equally important: Do the distractions cause “lapses in judgement, insensitivity to changing clinical conditions and medication errors.”1 If so, there are practical solutions that can help refocus our attention.

Kissler et al. offer several recommendations.  Initially, we need to recognize the difference between reachability and deep interpersonal availability. Most clinicians want to be reachable to help solve problems within their scope of practice, but spreading oneself too thin can jeopardize one's communication quality. Designing the physical spaces in which we interact with patients and colleagues is another area where we can build better attentiveness. For many years, the business and tech worlds believed that bullpens and shared office space would foster communication. Still, we are now beginning to realize that all the distractions may impede creativity and productivity. Finally, the University of Colorado team emphasizes the need to build attentiveness into one’s organizational culture: “Provide clinicians with the tools and language to prioritize attention in their daily practice.” That can be accomplished by developing a culture that encourages staffers to listen with curiosity, communicate with empathy, and remain open to others' perspectives, even when that perspective contradicts our understanding of the facts.

Of course, as every clinician knows, even the most attentive listener can still miss things. The medical interview can only uncover so much, necessitating a careful physical exam and diagnostic testing when appropriate. While imaging studies have always been a part of the diagnostic process, machine learning has taken these procedures to a new level, with companies like Zebra Medical, GE, Siemens, and AIDOC introducing useful services. AIDOC, for instance, has created a suite of services that combines three layers: an algorithmic layer, a product layer, and a clinically viable solution layer. All three are combined and implemented directly into the workflow. According to AIDOC, the platform reduces “turnaround time and increases quality and efficiency by flagging acute anomalies in real-time. Radiologists benefit from state-of-the-art deep learning technology that is "Always -on," running behind the scenes and freeing them to focus on the diagnosis itself.”

Surveys suggest a need for an always-on service to help radiologists cope with the unrealistic workload that many face daily. One study found that: “ Based on 255 uninterrupted eight-hour workdays per year, radiologists need to review one image every three to four seconds to meet workload demands.” The hectic pace likely contributes to misdiagnoses and loss of life. The diagnosis of lung cancer with imaging is one of the most challenging issues to contend with. It is estimated that misinterpreted chest X-rays are responsible for 90% of presumed errors in diagnosing pulmonary tumors.2  Mounting evidence suggests that ML-enhanced imaging data analysis may catch the disease at a much earlier stage, reduce hospital length of stay and health care costs, and save lives. For example, a prospective, randomized clinical trial that evaluated AIDOC-assisted CT scanning during the intracranial hemorrhage diagnosis found that algorithms gave clinicians an earlier heads-up.  Specifically, the researchers looked at 620 consecutive head CT scans. They collected the turnaround times (TAT) for positive ICH findings, i.e., how long it took to complete the CT scan to report the findings to clinicians who needed the results to make a treatment decision.3 Wismuller and Stockmaster compared TAT when CT results were flagged in radiologists’ worklists to CT results that were not flagged. When radiologists were told about the potentially dangerous findings early on, TAT was 73 minutes +/- 143 minutes, compared to 132 minutes +/- 193 minutes when they were left in the dark early on.

These prospective results were supported by retrospective analysis of a much larger data set. A study presented at the 2019 Society of Photo-Optical Instrumentation Engineers conference analyzed over 7,000 head CT scans from urban academic and trauma centers. Using convolutional neural networks, AIDOC generated a specificity of  99%, the sensitivity of 95%, and overall accuracy of 98% in diagnosing intracranial bleeds when compared to ground truth from expert neuroradiologists.4

Reports like this certainly don't imply that machine learning-enhanced algorithms will someday replace physicians. High-quality patient care will always require clinicians who are empathetic listeners. Nor do they suggest that AI will replace experienced radiologists. But they suggest that those who ignore digital medicine innovations will eventually be replaced by those willing to combine traditional approaches with emerging digital techniques that augment human decision-making.



1. Kissler MJ, Kissler K, Burden M. Toward  medical “ecology of attention.” New Engl J Med. 2021; 384: 299-301.

2. Del Ciello A, Franchi P, Contegiacomo A et al. Missed lung cancer: when, where, and why? Diagn. Interv Radiol. 2017;23:118-126.

3. Wismuller A, Stockmaster L. A Prospective Randomized Clinical Trial for Measuring Radiology Study Reporting Time on Artificial Intelligence-Based Detection of Intracranial Hemorrhage in Emergent Care Head CT. Presentation at SPIE Medical Imaging 2020 Conference, Houston, TX, February 15-20, 2020. https://arxiv.org/pdf/2002.12515.pdf

4. Ojeda P, Zawaideh M, Mossa-Basha M, et al. The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies. Proceedings SPID Medical Imaging 2019. https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10949/2513167/The-utility-of-deep-learning--evaluation-of-a-convolutional/10.1117/12.2513167.short

Monday, February 22, 2021

Diabetes Meets Machine Learning, Part 1

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

Of all the disorders that have responded well to artificial intelligence and machine learning, diabetes mellitus probably tops the list. The evidence supporting a role for Machine Learning (ML)-enhanced algorithms in managing the disease is persuasive and applies to several components of patient care, including screening, diagnosis, treatment, and prognosis.

Let's start with screening: As most clinicians know, there's a difference between screening for disease and diagnosing it. The former casts a much wider net, which means it will include more false positives. Once this larger cohort has been identified, more precise diagnostic testing can be performed to pinpoint patients who have the disorder.  As we explain in our next book, The Digital Reconstruction of Healthcare, screening tools for diabetes have been in existence for many years, including tools to help clinicians and patients identify the presence of prediabetes. Still, new technology has taken the screening process to a new level.

The American Diabetes Association (ADA) defines prediabetes as fasting plasma glucose between 100 and 126 mg/dl, a 2-hour oral glucose tolerance test reading between 140 and 200 mg/dl, or hemoglobin A1c of 5.7% to 6.5%.1 The Association has developed a risk assessment tool to help clinicians and patients; it asks for a patient's age, gender, family history of diabetes, the presence of hypertension, whether the patient is physically active, their ethnicity, and height and weight. Although this kind of scoring system will detect many persons with prediabetes, it has its limitations and ignores many changes in a patient's medical history over time. An artificial intelligence (AI)-enabled assessment tool that takes advantage of gradient boosting, on the other hand, is capable of factoring in many more risk factors gleaned from a patient’s electronic health record. This type of assessment system can accurately predict which patients will progress to full-blown diabetes, with an area under the curve (AUC) of 0.865.2 The gradient boosting approach has proven more accurate than logistic regression and evaluation of standard clinical cutoffs of blood glucose ≥ 110 mg/dl and HbA1c ≥ 6.0%. An analysis by Avivit Cahn from the Hebrew University of Jerusalem and her colleagues found the said ML algorithm incorporated 69 variables to achieve its superior AUC.

There is also reason to believe that AI-enhanced algorithms can help identify individuals before they become prediabetic. The Centers for Disease Control and Prevention (CDC) has a screening test to help predict prediabetes that asks one’s age, family history of diabetes, the existence of hypertension, race or ethnicity, whether one is physically active, gender, height and weight.

But logistic regression, artificial neural networks, random forest analysis, and gradient boosting have uncovered numerous other risk factors to predict prediabetes. Among the predictors that may help refine the risk stratification process: food security, citizenship, monocyte count, uric acid level, ALT, RBC count, and serum calcium and potassium.3 Whereas the CDC screening tool returned an area under the receiver operating characteristic (AUROC) of 64.4% in a large cohort from the National Health and Nutrition Survey, several AI-based algorithms generated AUROCs at or above 70%.

Several research projects suggest machine learning can improve the treatment of diabetes. In our book on mobile medicine, we discussed DreaMed,*  an Israeli vendor that developed DreaMed Diabetes to provide clinical decision assistance to clinicians managing Type 1 diabetes in patients who are using an insulin pump and continuous glucose monitoring (CGM). Its Advisor Pro algorithms collect several data types, including the patient’s daily blood glucose readings, meal carbohydrate data, insulin dosing records, and physical activity level. The software then uses event-driven machine learning and adaptive technology to create a more individualized regimen, adjusting its insulin dosing recommendations and offering behavior-modification recommendations accordingly. DreaMed Diabetes relies on a clinical AI program called MD-Logic. It uses fuzzy logic and adaptive learning algorithms to approximate how diabetes specialists reason as they adjust an individual's insulin regimen. The company claims that the software provides “faster analysis and deeper insights” than would normally be available with routine endocrinologist care. There is research support for this automated approach to Type 1 diabetes management.4

This coordinated interplay between an insulin pump and CGM sometimes referred to as closed-loop control or an artificial pancreas, has also been a focus of Mayo Clinic researchers. Yogish Kudva, MBBS, along with colleagues Sandeep Gupta and Ayan Bannerjee from Arizona State University, has been analyzing continuous glucose monitoring data from people with type 1 diabetes on the Medtronic MiniMed 670G closed-loop system* to develop software that would help each patient and clinic provider improve closed-loop control. They have also developed a system for measuring heart rate, respiratory rate, and self-report symptoms of viral infections in individuals with diabetes. The aim is to investigate the role of glycemic control on the progression of signs and symptoms of infectious disease in both at-home and hospital settings.

Similarly, Dr. Kudva participated in a recent investigation on the role of a closed-loop system to improve glycemic control in adolescents and young adults, part of the International Diabetes Closed-Loop Trial.5

Unfortunately, until recently, CGM has been out of reach for most Type 2 patients. But with the introduction of new, state-of-the-art technology, it is now available to them as well. We’ll explore these innovative solutions in part 2 of our series on machine learning-enhanced diabetes.

*Products and services mentioned are not endorsements.  Mayo Clinic has no relationship with any of these vendors.



1. American Diabetes Association. Classification and diagnosis of diabetes: standards of medical care in diabetes 2019. Diabetes Care. 2019;42(Suppl. 1):S13–S28. doi .org/10.2337/dc19-S002

2.  Cahn A, Shoshan A, Sagiv T el al. Prediction of progression from pre-diabetes to diabetes: development and validation of a machine learning model. Diabetes Metab Res Rev. 2019; e3252. doi.org/10.1002/dmrr.3252

3. De Silva K, Jonsson D, Demmer RT. A combined strategy of feature selection and machine learning to identify predictors of prediabetes. JAMIA. 2020;27: 396–406.

4. Nimri R, Muller I, Atlas E, et al. MD-logic overnight control for 6 weeks of home use in patients with type 1 diabetes: randomized crossover trial. Diabetes Care. 2014;37:3025_32.

5. Isganaitis E, Raghinaru D, Amber-Osborn L et al. Closed-Loop Insulin Therapy Improves Glycemic Control in Adolescents and Young Adults: Outcomes from the International Diabetes Closed-Loop Trial. Diabetes Technol. Ther. 2021l Jan 21. doi: 10.1089/dia.2020.0572.  Online ahead of print.

Friday, February 12, 2021

High-Quality Hospital Care — Minus the Hospital

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

In a time of COVID, the current care models just can’t remain in place. We need innovative ways to address the high cost of acute care. One solution that is taking a front seat is telemedicine.  Telecare has accelerated from 3-4% of visits in January 2020 to 90% in April to a new normal of 20% in 2021. Cultural change has modified patient expectations for the care they can get at a distance, which can be more convenient, less likely to result in COVID exposure, and more patient-centric.

Research has demonstrated that hospital-at-home programs for patients with specific acute medical conditions can reduce complications and reduce the cost of care by 30% or more.[1] One of the most progressive programs to focus on this transition was spearheaded by Johns Hopkins Hospital in 1994. Bruce Leff, MD, and his colleagues have tested this program with 455 elderly patients from three Medicare-managed systems and a VA medical center.[2] They found that the home model met the quality of care standards comparable to those expected of in-hospital programs. Also, "On an intention-to-treat basis, patients treated in hospital-at-home had a shorter length of stay (3.2 vs. 4.9 days) (P =0.004), and there was some evidence that they also had fewer complications. The mean cost was lower for hospital-at-home care than for acute hospital care ($5081 vs. $7480) (P < 0.001).”

A meta-analysis of 61 randomized clinical trials that looked at hospital-at-home projects found that among 42 trials, which included almost 7,000 patients, this approach reduced mortality (odds ratio, 0.81). Similarly, they decreased readmission rates by 25% (odds ratio, 0.75) and lowered costs. The same analysis revealed that treating every 50 patients in such a program saved one life.[3] Realizing the potential advantages of hospital-at-home programs, several large U.S. provider organizations have entered this space in the last few years, including Mayo Clinic, Partners Healthcare/Brigham and Women’s Hospital, and Mount Sinai Health System in New York. Across the globe, there are also significant programs in Australia, South Wales, and Spain.

Advanced Care at Home, a partnership between Mayo Clinic and Medically Home., tracks heart rate, blood pressure, pulse oximetry, temperature, and respiratory rate in its patient population, using Bluetooth-enabled devices wirelessly connected to the Mayo/Medically Home system. It also uses tablets, a back-up battery system and a Wi-Fi phone. There are, however, critical differences between many home-care programs and the Mayo Clinic system. Many hospital-at-home programs are targeted and designed for low-acuity hospital patients. They use physician house calls as the clinical delivery model. They have a short patient engagement period (2-4 days).

The Medically Home affiliated setup is designed to handle an extended length of stay that includes acute, post-acute and preventative care. It uses a scalable “decentralized” model for high-acuity care and can manage a broad set of diverse use cases and support an extensive patient census. The program uses screening, training, contracting, quality management, and technology and converts "post-acute" community-based supply providers into acute-level providers, bringing goods and services to high-acuity patients at home while focusing heavily on the role of paramedics as the centerpiece of its ability to provide rapid-response capabilities. In practical terms, that means paramedics and other providers go into the home while being virtually connected with a centralized medical command center staffed by physicians who guide the care for decentralized patients and the decentralized providers that care for them.

Advanced Care at Home has made measurable progress within a relatively short period, going from a speculative pilot project about a year ago to a business plan that will likely prove profitable in 2022. During a recent Zoom call with Ajani (AJ) Dunn, Administrator for the program, he emphasized, “It’s a story about volume.  As we looked at the model, we asked ourselves: Will it be effective? And we found that by scaling it up to the point where we can take out the fixed cost of a traditional hospital stay and replace that with the small variable costs of each service we deliver in the home, we can have a sustainable program.” Dunn explained that by working through a centralized command center staffed by physicians and getting buy-in from third-party payors, this approach is slowly turning the corner financially. Since most insurers do not have a hospital at home plan built into their policy, that has required the Mayo team to negotiate with payors one by one, explaining the cost-effectiveness of the new model.

The next step in the program’s growth will require finding ways to reinvent the existing system. “The off-the-shelf system we use to administer Advanced Care at Home is well-calibrated for traditional medical admissions, including COPD, CHF, pneumonia and the like. That approach involves drawing labs, titrating medications, etc. But we need other ways to use the system. Our plan now is to embed teams into other disease states, including oncology, cardiovascular disease, and transplantation, to understand the natural progression of each disease and the necessary interventions. Then we can create clinical protocols and administrative logistics that replicate in-hospital care in the home, but customized for each disease state.”

Undoubtedly, concerns about high costs and unexpected complications will continue to dissuade patients from seeking in-hospital care for many years. But as this new model expands, it's likely more patients will see the advantages of seeking high-quality hospital care — minus the hospital.



1. Klein S. “Hospital at home” programs improve outcomes, lower costs but face resistance from providers and payers. The Commonwealth Fund. Accessed January 15, 2020. https://www.commonwealthfund.org/publications/newsletter-article/hospital-home-programs-improve-outcomes-lower-costs-face-resistance

2. Leff B, Burton L, Mader SL, et al. Hospital at home: feasibility and outcomes of a program to provide hospital-level care at home for acutely ill older patients. Ann Intern Med. 2005;143:798–808.

3. Caplan GA, Sulaiman NS, Mangin DA, et al. A meta-analysis of “hospital in the home.” Med J Aust. 2012;197:512–519.

Wednesday, February 10, 2021

Let Your Freak Flag Fly

 John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

If Mary Putnam Jacobi were alive today, she would probably embrace artificial intelligence (AI), machine learning, and data analytics. Dr. Jacobi might best be described as the mother of modern scientific medicine  or at the very least one of its founding parents. In 1868, she was the first woman to enroll in the University of Paris School of Medicine. After graduating in 1871, this unconventional thinker arrived in the U.S., where she advocated for the inclusion of laboratory science, experimentation and statistics as the foundation cornerstone of modern medical practice. Equally important, Jacobi “became a powerful advocate for the equal contribution of women to medicine.” Pushing clinicians to buy into the notion that experimentation and statistics were needed for good quality patient care may seem unimpressive today. Still, it was almost heresy in an age when the received wisdom from one’s medical school professor was all that was necessary to “demonstrate” that a treatment protocol was effective. Given her “color-outside-the-lines” approach, it’s not hard to imagine her becoming passionate about the role of AI in medicine — not as a panacea but as an indispensable adjunct to human reasoning and clinical trials.

The evidence shows that machine learning, advanced data analytics, and subgroup analysis all have the power to reinvent the way patient care is delivered and the potential to personalize that care. But as we said in an earlier blog, while there are several well-documented AI-based algorithms now available, there are also several questionable vendors that have rushed to market with little evidence to support their claims. This dilemma begs the question: How do we ensure that well-supported digital tools get the attention they deserve? Several new guidelines have been published to encourage more robust AI-related research, but clinicians also need to develop machine learning literacy. Put another way, we can all benefit from gaining a better grasp of what’s under the hood, explained in plain, jargon-free English. That was one of the goals of our last book, Reinventing Clinical Decision Support.

Our research demonstrates that clinical decision support certainly needs to be reinvented. If you look back over the last several decades, you’ll find that hundreds of clinical research studies have been misinterpreted. A review of 71 randomized clinical trials in 1987, for instance, concluded that numerous treatments were useless; a closer analysis, however,  found that all 71 had generated false-negative results because the populations they studied were too small — a type 2 statistical error. [1] Fast forward to 1994; a JAMA analysis revealed that 383 RCTs, many published in the world’s top academic journals, likewise jumped to the conclusion that several treatments were ineffective, once again because the trials had enrolled too few patients. [2]

More recently, there’s reason to believe that the bedrock upon which many day-to-day clinical decisions rest is somewhat shaky. It’s a foundation that can be made stronger with ML-based tools like convolutional neural networks, random forest modeling, gradient boosting, and clustering. A case in point: The Look AHEAD study that published in 2013.[3] This RCT assigned over 5,000 overweight patients with type 2 diabetes to either an intensive lifestyle modification program or a control group. The investigators' goal was to determine if the lifestyle program would reduce the incidence of cardiovascular events. The study was terminated early because there were no significant cardiovascular differences between the intervention and control groups.

An ML-fueled analysis that used random forest modeling to re-examine the Look AHEAD data turned these results upside down. During random forest analysis, a series of decision trees are created. Initially, the technique randomly splits all the available data — in this case, the stored characteristics of about 5,000 patients in the Look AHEAD study — into two halves. The first half serves as a training data set to generate hypotheses and construct the decision trees. The second half of the data serves as the testing data set.

Using this technique, Baum et al. constructed a forest that contained 1,000 decision trees. They looked at 84 risk factors that may have been influencing patients’ response or lack of response to the intensive lifestyle modification program, including numerous characteristics that researchers rarely, if ever, consider when doing a subgroup analysis. The random forest modeling also allowed the investigators to examine how these variables interact in multiple combinations to impact clinical outcomes. In the final analysis, Baum et al.[4] discovered that intensive lifestyle modification did prevent cardiovascular events for two subgroups, patients with HbA1c 6.8% or higher (poorly managed diabetes) and patients with well-controlled diabetes (Hba1c < 6.8%) and good self-reported health. That finding applied to 85% of the entire patient population studied. On the other hand, the remaining 15% who had controlled diabetes but poor self-reported general health responded negatively to the lifestyle modification regimen. The negative and positive responders canceled each other out in the original Look AHEAD statistical analysis, falsely concluding that lifestyle modification was useless.

AI-enhanced data analyses like this only serve to reinforce Maru Putnam Jacobi’s contention that coloring outside the lines will propel health care in new directions. Her unconventional mindset, and ours, are best summed up in the words of Crosby Still and Nash: “Let your freak flag fly!”


1. Freiman JA et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. N Engl J Med. 1978; 299:690-694.

2Moher D et al. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA. 1994; 272:122.

3. Look AHEAD Research Group. Cardiovascular effects of intensive lifestyle intervention in type 2 diabetes. N Engl J Med. 2013. 369:145-154.

4. Baum, A et al. Targeting Weight Loss Interventions to Reduce Cardiovascular Complications of Type 2 Diabetes: A Machine Learning-Based Post-Hoc Analysis of Heterogeneous Treatment Effects in the Look AHEAD Trial. Lancet Diabetes Endocrinology, 2017;  5: 808–815.

Thursday, February 4, 2021

The Therapeutic Potential of Voice Technology

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

The human voice is capable of extraordinary feats of genius and everyday acts of kindness.  It can recite Shakespearean sonnets, teach our children moral values, stir audiences with a dramatic performance, and much more. But few of us ever imagined it capable of assisting in the diagnosis of disease. That’s about the change, as evidenced by several innovative projects in voice technology.

With the right digital tools, it is now possible to analyze a patient’s speech patterns to detect clues to underlying pathological issues. Elad Maor, MD, Ph.D., with the Mayo Clinic Department of Cardiovascular Medicine, and his colleagues have looked at voice samples from about 100 patients who underwent coronary angiograms, asking them to read text excerpts and respond to questions about positive and negative emotional experiences. Their recorded responses found subtle differences in vocal pitch and intensity between patients who were ultimately diagnosed with heart disease and normal controls. [1] Dr. Maor and his associates concluded: “One possible explanation for our interesting finding is the documented association between mental stress, the adrenergic system, and voice. . . . Emotional stress conditions change the human voice, including an increase in fundamental frequency. . . . [O]ne possible hypothesis to interpret our findings is that the association between voice and atherosclerosis is mediated by hypersensitivity of the adrenergic system to stress. The association between stress, the adrenergic system, and atherosclerosis is well established on the basis of robust data.”

There is also reason to believe that voice technology may help detect pulmonary hypertension. Jaskanwal Deep Singh Sara, M.B., Ch.B. (also with the Mayo Clinic), in collaboration with scientists from Vocalis Health (an Israeli vendor), has analyzed voice recordings among patients who had invasive cardiac hemodynamic testing, the standard approach to diagnosing pulmonary hypertension. The recordings were analyzed to measure pitch, loudness, jitter, and other metrics. Dr. Sara and his colleagues found a significant association between an invasively derived hemodynamic index used to measure pulmonary hypertension and the vocal biomarkers. Patients with pulmonary arterial pressure at or above 35 mmHg had higher mean vocal biomarker readings than those with pressure readings at or below 35 mmHg. Given that invasive testing occurred during cardiac catheterization, the non-invasive collection of a patient's voice patterns holds promise as a safer alternative. If controlled clinical trials confirm the findings, it will likely reduce the cost and risk associated with pulmonary hypertension (PH) diagnosis. [2]

Investigators in Belgium also have had success using vocal characteristics as part of a suite of tools to help detect Parkinson's Disease in its early stages. [3]  Some studies suggest that 60-90% of patients have subtle changes in their voice and speech patterns when initially diagnosed. In situations in which the diagnosis is questionable, however, a neurologist might administer a loading dose of levodopa, one of the standard drugs used to treat the disease and watch for improvements in a patient’s speech and voice. After administering the drug, the Flemish researchers monitored the strength of a patient's facial or mouth muscles and evaluated vocal quality, frequency, breathiness, phonation time — how long a person can sustain a vocal sound on one deep breath — and several other parameters, folding them into a metric called a Voice Handicap Index. They found that these markers helped distinguish patients with idiopathic Parkinson's disease from healthy individuals.

Mayo Clinic is also exploring the value of voice technology, coupled with artificial intelligence, to address patients' needs with neurological disorders and related motor speech disorders. The project's goals include creating digital tools for voice-based disease detection in the office, over the phone, and in a patient's home; earlier, more accurate and more holistic diagnoses; and the provision of individualized, in-home markers of disease progression and treatment response. To accomplish those goals, the Neurology AI Program and other Mayo researchers and clinicians are developing a fully automated digital speech diagnostics platform that can take a patient's speech sample and provide probabilistic diagnoses. Those taking the lead in this initiative believe that the primary goal should not be to map directly onto a clinical diagnostic label but instead use AI to extract clinically meaningful information from a speech sample. 

Mayo Clinic-based speech pathologists Darley, Aronson and Brown [4] proposed a similar idea in 1969 in their seminal work on the dysarthrias, speech disorders caused by muscle weakness.  While this theoretical construct was previously based on conventional speech labels, advances in AI have made it feasible to build such feature space using latent patterns in the data, which can then be labeled and linked to dense, multivariate medical data. This Deeply Annotated Speech Latent space can then be used to characterize new speech samples. When connected to non-speech medical and demographic data, a broader neurologic or systemic diagnosis can be rendered. The model will likely have the most considerable impact if deployed on smartphone devices, smart speakers or other wearable technology.

Voice technology has yet to surpass Shakespeare’s genius, replace a gifted actor's heart-wrenching performance,  or serve as a moral compass for the next generation. Still, the evidence suggests it is ushering in a new generation of intelligent digital tools that will likely transform patient care. 


1. Maor E, Sara JD, Orbelo DM, et al. Voice signal characteristics are independently associated with coronary artery disease. Mayo Clinic Proceedings. 2018;93:840–847.

2. Sara JDS, Maor E, Borlaug B, et al. Non-invasive vocal biomarker is associated with pulmonary hypertension. PLOS ONE. 15(4):e0231441. doi.org/10.1371/journal.pone.0231441

3. Lechien JR, Delsaut B, Abderrakib A et al. Orofacial Strength and Voice Quality as Outcome of Levodopa Challenge Test in Parkinson Disease. Laryngoscope, 130:E896–E903, 2020.

4. Frederic L Darley, Arnold E Aronson, and Joe R Brown. Clusters of deviant speech dimensions in the dysarthrias. Journal of speech and hearing research, 12(3):462–496, 1969.

Tuesday, January 26, 2021

Wearable Danger

This article is written by John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform.

If you ask health care executives what keeps them up at night, many would sum up their worries in one word: ransomware.  By one estimate, 56% of organizations suffered a ransomware attack in the last year. While there are countless ways in which a cyberthief can penetrate a facility’s computer network to block access to essential data, one avenue that gets too little attention is through wearables and related medical devices.  A growing number of providers are now allowing patients to send data from blood glucose monitors, blood pressure cuffs, bed sensors, and portal EKG devices to their networks. And during the COVID-19 pandemic, many more clinicians are working remotely using their own laptops, tablets and smartphones to access a hospital or office EHR system. All these connections are potential opportunities for hackers to infiltrate your computer network. And the word potential doesn’t fully capture the danger.

In 2019, for instance, FDA issued an alert to health professionals warning about a cybersecurity vulnerability affecting Medtronic implantable cardiac devices (ICDs), programmers and home monitors. The agency found the vulnerability in the wireless telemetry technology used to communicate between the ICDs, clinic programmers and home monitors. Similarly, the company that makes the OneTouch insulin pumps contacted patients using the device of the possibility that it could be hacked and reprogrammed, which could have life-threatening consequences.

During a recent conversation with Leon Lerman, CEO of Cynerio, a cybersecurity solutions firm, he explained that once a hacker infiltrates a computer network, often through a phishing scam and malware, medical devices become easy targets. That’s the case for several reasons, including inadequate segmentation and outdated operating system. Virtual local access networks (VLANs) are one way to address the issue because they limit the number of users allowed to have access to a specific part of the network.  Unfortunately, a study sponsored by Forescout, a security firm, found “only 49 percent of medical devices were deployed across 10 virtual local access networks (VLANs) or fewer ….”

Outdated operating systems, an easy access point for hackers, remain a persistent problem for health care providers, as are outdated software applications. An international survey involving 600 health care IT professionals in 2019 found more than 1 out of 4 organizations were still running Windows 7 on their medical devices. The danger posed by this practice may not be immediately obvious to most clinicians, but because many older OSs are no longer supported by their manufacturers, security patches are no longer available to block newly designed digital threats. Of course, health care providers running currently supported operating systems can also fall victim to cyberattacks if they fail to install security updates as soon as they become available. That’s how the infamous WannaCry ransomware worm was able to penetrate the NHS and numerous other networks; it affected more than 200,000 computers worldwide in 150 countries. Microsoft had already issued a security patch before the WannaCry incident, but many organizations had neglected to install it in time.

One of the challenges in keeping operating systems up to date is the restrictions that hospital IT teams face when they try to address the issue. Most devices are black boxes in the sense that the manufacturer does not allow users to touch the software; doing so without the company’s permission usually voids the warranty. That makes it virtually impossible for a hospital or medical practice to install security updates to legacy OSs, even when they are available. If the device manufacturer is cooperative, it may be possible to have their technicians do these updates. When that’s not an option, segmentation becomes all the more important.

Fortunately, many device manufacturers are now beginning to realize that their reputations depend upon developing machinery that is not just clinically functional but hardened to cyberattacks. Many new devices come with a Manufacturer Disclosure Statement for Medical Device Security (MDS2) that spells out the security protocols used on the device, whether anti-malware software has been installed, and whether it should even be connected to the Internet.

The adage about necessity being the mother of invention certainly applies to the Internet of Medical Things. As the health care ecosystem experiences more cyberattacks, we learn to adapt, and out of necessity, develop creative tools to defend our networks and most importantly we learn more effective ways to protect our patients—our number one priority.

Friday, January 22, 2021

How is AI Impacting Health Care Today?

By John Halamka and Paul Cerrato*

We are often asked this question during interviews, podcasts and speaking engagements. It’s a complicated question that requires context. A closer look at the research and product offerings in digital health demonstrates that there are several high-quality, well-documented algorithms now available, but there are also several questionable vendors that have rushed to market with little evidence to support their claims. Separating the wheat from the chaff can be a full-time occupation.

We recently received a press release from a large U.S. company highlighting a new AI system that may be able to diagnose dementia using a patient’s vocal patterns. The vendor pointed out that its research generated an area under the curve (AUC) of 0.74 for its system, which suggests that at least one in 4 patients with dementia would be overlooked. With these concerns in mind, the question is: What kind of guidelines can clinicians and technologists on the front lines turn to when they want to make informed choices?

Ideally, we need an impartial referee that can act as a Consumer Reports type service, weighing the strengths and weaknesses of AI offerings, with brief summaries of the evidence upon which they base its conclusions. In lieu of that, there are criteria that stakeholders can tap to help with the decision making process. As we point out in a recent review in the New England Journal of Medicine journal, called NEJM Catalyst, at the very least, there should be prospective studies to support any diagnostic or therapeutic claims. Too many algorithms continue to rely on retrospective analysis to support their products. (1) In our NEJM analysis, we include an appendix entitled “Randomized Controlled Trials and Prospective Studies on AI and Machine Learning,” which lists only 5 randomized controlled trials and 9 prospective, non-RCT studies. When one compares that to the thousands of AI services and products coming to market, it’s obvious that digital health still has a long journey to make before it’s fully validated.

That’s not to suggest that there are no useful, innovative AI and machine learning tools that are well supported, as well as several that are coming through the pipeline.  There are credible digital tools to estimate a patient’s risk of colorectal cancer (ColonFlag), manage type 1 diabetes (DreaMD), and screen for diabetic retinopathy (IDx), all of which are supported by good evidence.** The FDA has also published a database of approved AI/ML-based medical technologies, summarized by Stan Benjamens and his associates in npj Digital Medicine.(2) (Keep in mind when reviewing this database, however, that some of the algorithms cleared by FDA were based on very small numbers of patients.)

A recent virtual summit gathered several thought leaders in AI, digital health and clinical decision support to create a list of principles by which such tools can be judged. Spearheaded by Roche and Galen /Atlantica, a management consulting firm, its summit communique refers to the project as “A multi-stakeholder initiative to advance non-regulatory approaches to CDS quality.”  Emphasizing the need for better evidence, the communique states: “The development of CDS is driven by increasing access to electronic health care data and advancing analytical capabilities, including artificial intelligence and machine learning (AI/ML). Measures to ensure the quality of CDS systems, and that high-quality CDS can be shared across users, have not kept pace. This has led some corners of the market for CDS to be characterized by uneven quality, a situation participant likened to “the Wild West.”

The thought leaders who gathered for the CDS summit certainly aren’t the only ones interested in improving the quality of AI/ML-enhanced algorithms.  The SPIRIT AI and CONSORT-Ai Initiative, an international collaborative group that aims to improve the way AI-related research is conducted and reported in the medical literature, has published 2 sets of guidelines to address the issues we mentioned above. The twin guidelines have been published by Nature Medicine, BMJ and Lancet Digital Health. (3,4) They are also available on the group’s web site.

With all these thought leaders and experts on board, there’s no doubt the AI ecosystem is gradually transitioning from the “Wild West” into a set of well-defined and repeatable processes that health care stakeholders can trust. http://geekdoctor.blogspot.com/2021/01/to-build-fire.html


*Paul Cerrato is a senior research analyst and communications specialist at Mayo Clinic Platform

 **Products mentioned are not endorsements.  Mayo Clinic has no relationship with any of these vendors.



1. Halamka J, Cerrato, P. The Digital Reconstruction of Health Care. NEJM Catalyst: Innovations in Care Delivery. Vol 1 (6); Nov-Dec 2020.

2. Benjamens S, Dhunnoo P, Mesko B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. Npj Digital Medicine. 3:118, 2020. https://www.nature.com/articles/s41746-020-00324-0

3. Cruz Rivera, S, Liu, X, Chan, A-W et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nature Medicine volume 26pages13511363(2020)

4. Liu, X, Cruz Rivera, S, Moher D et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension Nature Medicine volume 26pages13641374(2020)

Tuesday, January 19, 2021

Addressing racism with compassion, data analytics

This article is written by John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform.

We speak often about the need to combine human and artificial intelligence (AI) to improve patient care. Equally important is the marriage of compassion and data analytics ― a powerful duo that is proving invaluable in the battle to eradicate the systemic racism that still permeates health care.

Unfortunately, numerous examples demonstrate that systemic racism continues to affect the patient experience and leads to mistrust of health care institutions among people of color.

Some of us are familiar with the unethical Tuskegee syphilis study, in which the U.S. Public Health Service observed a large population of African American men with untreated syphilis between 1932 and 1972. As part of that study, participants with syphilis were not informed of their diagnosis, nor treated for it. They were told they were receiving free health care from the federal government.

As another example of systemic racism in health care, a recent journal article in the Proceedings of the National Academy of Sciences of the United States of America revealed that among 1.8 million U.S. births, the Black newborn mortality rate was three times higher when white doctors delivered the child, compared to Black doctors.

Awareness of such prejudices causes fear of interacting with the health care system, as evidenced by a December 2020 survey by the Kaiser Family Foundation in which 35% of Black respondents said they definitely or probably would not get a COVID-19 vaccine.

The list of inequities associated with racism includes numerous other problems. Blacks have a lower average life expectancy and they are less likely to have been vaccinated, according to a 2015 review by the Department of Health and Human Services. A 2015 review by Paradies et al also found that racism was "associated with poorer mental health, including depression, anxiety, psychological stress and various other outcomes … and with poorer general health.

Since the Black Lives Matter movement came to prominence in the U.S., many health care leaders have spoken out to address these disparities. Mayo Clinic has likewise taken a strong position on the subject and has put skin in the game by investing in a 10-year, $100 million effort to eradicate racism in all its forms.

That effort will include initiatives to:

  • Increase recruitment of researchers and clinical trial patients from underrepresented racial and ethnic groups.
  • Find ways to recruit and retain physicians, nurses and supervisors from underrepresented groups.
  • Build out its digital and telehealth technology to make patient care more equitable around the nation.

Mayo also is working on programs to increase its own patient population's diversity, with special attention paid to Black patients.

At the local level, Mayo recently awarded several grants to communities to advance racial equity. Specifically, its EverybodyIN Fund for Change has given grants to 36 organizations in Mayo Clinic communities, including 17 organizations in the regions Mayo Clinic Health system serves.

Mayo Clinic's core values are the springboard for these initiatives and programs. These values include respect for everyone in our diverse community, providing compassionate care with an emphasis on sensitivity and empathy, and integrity to the highest principles of professionalism.

Of course, Mayo realizes that values have to be accompanied by specific actions to have any lasting impact. In addition to the concrete actions mentioned above, we are completing the most extensive analysis of care disparities in the U.S. to guide our steps.

In conjunction with Change Healthcare's data scientists, Mayo is creating a novel national disparities road map using Change Healthcare's enormous amount of linked data containing social determinants of health and health care use. The data will enable us to identify geographic areas and patient populations to first target as we work to eliminate health care inequities. We aim to develop a series of specific health care delivery interventions, leveraging many of the digital tools that are now part of the Mayo Clinic Platform that can address patient-level factors and structural factors that have contributed to, and continue to contribute to, racism in medicine.

We also have created an AI playbook that provides the tools and techniques needed to select unbiased data sets. This playbook can be used to develop algorithms to measure the biases in machine learning-based algorithms being used to deliver health care.

As we have mentioned in previous articles and books, several of the digital tools being recommended as diagnostic and therapeutic aids do not represent the populations they attempt to serve, leaving out adequate numbers of persons of color and other minorities.

In our book, "The Transformative Power of Mobile Medicine," we speak at length about the power of words, pointing out that they can persuade skeptics, overcome bigotry, wound colleagues, disrupt the status quo, ruin reputations, shatter misconceptions, deceive the uninformed, endear us to loved ones, and comfort the grief stricken.

Compassion is one of those powerful words, especially when it's backed by the actions of clinicians, executives and technologists who put patients first, regardless of the color of their skin.

Thursday, January 14, 2021

Responding to Misinformation

This article is written by John Halamka, M.D.president, Mayo Clinic Platform, and Paul Cerratosenior research analyst and communications specialist, Mayo Clinic Platform.

The singer/songwriter Paul Simon once penned the lyrics: A man hears what he wants to hear and disregards the rest (The Boxer). If that’s the case, how do we respond to misinformation that contradicts the data/evidence guiding development of treatment and cures?  If the only audience willing to read such articles are already critical thinkers, perhaps we are just preaching to the choir. And if by chance, a person who believes in controversial ideas does read articles based on real world evidence, will they consider them a one-sided discussion by the “medical-industrial establishment?” In our soon-to-be published book, The Digital Reconstruction of Healthcare (HIMSS/CRC Press), we discuss this dilemma at length. Here are a few highlights from that analysis. 

There is evidence to justify at least some mistrust among the public. For example, when physicians were asked what treatment recommendations they would make for patients and what decisions they would make for themselves if they were in similar circumstances, investigators found the clinicians would have made different choices for themselves: “Among those asked to consider our colon cancer scenario (n=242), 37.8% chose the treatment with a higher death rate for themselves but only 24.5% recommended this treatment to a hypothetical patient.’’(1) Even more concerningly are reports that suggest many medical procedures continue to be performed despite lack of strong scientific evidence supporting their efficacy. The problem has become significant enough to prompt the formation of the Right Care Alliance, “ a collaboration between health-care professionals and community groups that seeks to counter a trend: increasing medical costs without increasing patient benefits.”(2)

That misinformation conversation can be much shorter when health care leaders readily admit their mistakes and any uncertainty about the treatment protocols they are recommending. Chances are, such uncertainties are going to be revealed given the public's access to scientific data from clinical research that was once hidden from view.

Building the kind of trust that opens the minds of science skeptics also requires we accurately report the facts, theories and controversies when dealing with patients and the general public. This may seem an obvious weapon in the battle to debunk unscientific views, but it can be challenging for several reasons. Explanations take time and most clinicians are too busy discussing diagnosis and treatment in the short window of time they have with each patient to adequately counter misinformation. 

Another barrier to addressing misinformation is the nature of the scientific process itself. During the COVID-19 pandemic, for example, several critics have attacked statements by infectious disease specialists because their advice has changed over time as more data became available from a larger population of infected patients. The assertion, “The experts are constantly contradicting themselves,” reflects a lack of understanding of how the scientific method works, as well as the laws of probability.

When crafting a message that will reach skeptics, it also helps to understand the motivation that’s sometimes behind such doubts.  The motivation to believe falsehoods and half-truths, is complicated. Steven Pinker, a Harvard University professor of psychology, offers a plausible theory. Citing the research of legal scholar Dan Kahan, he points out that: “Certain beliefs become symbols of cultural allegiance. People affirm or deny these beliefs to express not what they know but what they are. We all identify with particular tribes or subcultures, each of which embraces a creed on what makes for a good life and how society should run its affairs.” (3) That observation implies that rejecting a deeply held belief is betraying one’s tribe, risking the loss of peer respect.

There is no magic pill to cure the misinformation epidemic we are currently experiencing. The problem has existed for centuries and is unlikely to disappear any time soon. Many believed in astrology in the 12th century just as they do now. Skeptics have likewise questioned the value of vaccinations for centuries. During the 1918 flu pandemic, some even claimed the disease was being spread as a result of a massive nationwide vaccine program. (4) Despite these myths, it is possible to overcome misconceptions with a combination of respect, patience and a willingness to admit that sometimes, the experts are wrong. Perhaps Paul Simon was too cynical. 


*Paul Cerrato is a senior research analyst and communication specialist at Mayo Clinic Platform 




1. Ubel PA, Angott AM, Zikmund-Fisher B. Physicians Recommend Different Treatments for Patients Than They Would Choose for Themselves. Arch Intern Med. 2011; 17: 639-634.

2. Epstein D, ProPublica. When Evidence Says No, but Doctors Say Yes. The Atlantic. Feb 22, 2017. https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/


3. Pinker S. Enlightenment Now. The Case for Reason, Science, Humanism and 

Progress. New York, NY: Viking; 2018:357. 


4. Reuters Staff. False claim: the 1918 influenza pandemic was caused by vaccines. Reuters. April 1, 2020.  

Monday, January 4, 2021

To Build a Fire

In 2021, much of our work at Mayo Clinic Platform will be creating repeatable processes that achieve their intended result in a timely, repeatable, scalable fashion. To understand what it means to achieve process maturity, let me tell the story of firewood management at Unity Farm Sanctuary, a great illustration of use case definition and attention to detail.

At the Sanctuary, we heat the farmhouse in the evening with a wood fire using fallen trees from the property. The logs must be sorted into wood species — ash and black birch can be burned without aging, while maple and oak must be aged. Cedar and pine are not good firewood because their oils cause the wood to pop and sputter. Poplar is not a good firewood because it smells and doesn't generate much heat.

Once we've identified the right wood for the right purpose, it needs to be cut into logs less than 2 feet long so they can be split and stored.

How do you cut up a fallen tree? You need multiple tools, including a chain saw for bucking, a forest axe for limbing, a timber jack to lift the tree off the ground, a sawbuck to trim the logs that are too long and a felling wedge to prevent the chain saw from getting pinched as logs are cut.*

If you have all these tools and the training to use them, you can reduce a fallen tree into firewood for splitting.

Then how do you split it? An engineer in West Bridgewater, Mass., custom built the SuperSplit, which uses flywheels instead of hydraulics. With this tool, I can split a cord of wood by myself in 30 minutes.

Then how do you store it? We use firewood brackets to create whatever size and shape storage we need, and then we cover it with a tarp.

Next, we have to transport it to the fireplace. We use a Vermont cart with flat-free tires and a log carrier.

When the chain saw chains are dull and covered with sap, how do you clean and sharpen them? We use a small tank of mineral spirits to soak the chains, and then scrub them with a stainless-steel brush. Then we use a commercial chain sharpener adjusted at precise angles to bring each chain back to factory specification. Then we use another tank to soak the chain in oil before storing it.

I mention all these process steps because to achieve maturity, we needed to identify each action to turn a fallen tree into a cozy fire. We had to implement the collection of technologies and training to do it rapidly and repeatedly.

In the Mayo Clinic Platform, we're developing mature processes for bringing on new partners, ingesting new kinds of data, launching new projects with existing joint ventures, evaluating AI algorithms and accelerating pilots with startups. Like the tree-to-fire procedure at Unity Farm Sanctuary, we’ll develop detailed use cases, appropriate vendors, training/staffing, toolsets and key performance indicators to support these Platform processes.

Maneesh Goyal, COO of Platform, notes that we need People, Processes and Products to be successful at Mayo Clinic Platform. My role as President of Mayo Clinic Platform and co-founder of Unity Farm Sanctuary is to support these concepts in both my professional and home life. 

*Products mentioned are not endorsements. The Sanctuary has no relationship with any of these companies.