Tuesday, March 2, 2021

Listen Better, See Deeper

Combining Medical Attentiveness with Artificial Intelligence

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

Embracing an “ecology of attention” will significantly improve patient care, according to Mark Kissler, MD, at the University of Colorado.1 Kissler and his colleagues point out that clinicians spend much of their time multi-tasking and navigating around interruptions. While such juggling acts are often unavailable, it’s important to occasionally step back and ask: Is this the best use of my time? Equally important: Do the distractions cause “lapses in judgement, insensitivity to changing clinical conditions and medication errors.”1 If so, there are practical solutions that can help refocus our attention.

Kissler et al. offer several recommendations.  Initially, we need to recognize the difference between reachability and deep interpersonal availability. Most clinicians want to be reachable to help solve problems within their scope of practice, but spreading oneself too thin can jeopardize one's communication quality. Designing the physical spaces in which we interact with patients and colleagues is another area where we can build better attentiveness. For many years, the business and tech worlds believed that bullpens and shared office space would foster communication. Still, we are now beginning to realize that all the distractions may impede creativity and productivity. Finally, the University of Colorado team emphasizes the need to build attentiveness into one’s organizational culture: “Provide clinicians with the tools and language to prioritize attention in their daily practice.” That can be accomplished by developing a culture that encourages staffers to listen with curiosity, communicate with empathy, and remain open to others' perspectives, even when that perspective contradicts our understanding of the facts.

Of course, as every clinician knows, even the most attentive listener can still miss things. The medical interview can only uncover so much, necessitating a careful physical exam and diagnostic testing when appropriate. While imaging studies have always been a part of the diagnostic process, machine learning has taken these procedures to a new level, with companies like Zebra Medical, GE, Siemens, and AIDOC introducing useful services. AIDOC, for instance, has created a suite of services that combines three layers: an algorithmic layer, a product layer, and a clinically viable solution layer. All three are combined and implemented directly into the workflow. According to AIDOC, the platform reduces “turnaround time and increases quality and efficiency by flagging acute anomalies in real-time. Radiologists benefit from state-of-the-art deep learning technology that is "Always -on," running behind the scenes and freeing them to focus on the diagnosis itself.”

Surveys suggest a need for an always-on service to help radiologists cope with the unrealistic workload that many face daily. One study found that: “ Based on 255 uninterrupted eight-hour workdays per year, radiologists need to review one image every three to four seconds to meet workload demands.” The hectic pace likely contributes to misdiagnoses and loss of life. The diagnosis of lung cancer with imaging is one of the most challenging issues to contend with. It is estimated that misinterpreted chest X-rays are responsible for 90% of presumed errors in diagnosing pulmonary tumors.2  Mounting evidence suggests that ML-enhanced imaging data analysis may catch the disease at a much earlier stage, reduce hospital length of stay and health care costs, and save lives. For example, a prospective, randomized clinical trial that evaluated AIDOC-assisted CT scanning during the intracranial hemorrhage diagnosis found that algorithms gave clinicians an earlier heads-up.  Specifically, the researchers looked at 620 consecutive head CT scans. They collected the turnaround times (TAT) for positive ICH findings, i.e., how long it took to complete the CT scan to report the findings to clinicians who needed the results to make a treatment decision.3 Wismuller and Stockmaster compared TAT when CT results were flagged in radiologists’ worklists to CT results that were not flagged. When radiologists were told about the potentially dangerous findings early on, TAT was 73 minutes +/- 143 minutes, compared to 132 minutes +/- 193 minutes when they were left in the dark early on.

These prospective results were supported by retrospective analysis of a much larger data set. A study presented at the 2019 Society of Photo-Optical Instrumentation Engineers conference analyzed over 7,000 head CT scans from urban academic and trauma centers. Using convolutional neural networks, AIDOC generated a specificity of  99%, the sensitivity of 95%, and overall accuracy of 98% in diagnosing intracranial bleeds when compared to ground truth from expert neuroradiologists.4

Reports like this certainly don't imply that machine learning-enhanced algorithms will someday replace physicians. High-quality patient care will always require clinicians who are empathetic listeners. Nor do they suggest that AI will replace experienced radiologists. But they suggest that those who ignore digital medicine innovations will eventually be replaced by those willing to combine traditional approaches with emerging digital techniques that augment human decision-making.



1. Kissler MJ, Kissler K, Burden M. Toward  medical “ecology of attention.” New Engl J Med. 2021; 384: 299-301.

2. Del Ciello A, Franchi P, Contegiacomo A et al. Missed lung cancer: when, where, and why? Diagn. Interv Radiol. 2017;23:118-126.

3. Wismuller A, Stockmaster L. A Prospective Randomized Clinical Trial for Measuring Radiology Study Reporting Time on Artificial Intelligence-Based Detection of Intracranial Hemorrhage in Emergent Care Head CT. Presentation at SPIE Medical Imaging 2020 Conference, Houston, TX, February 15-20, 2020. https://arxiv.org/pdf/2002.12515.pdf

4. Ojeda P, Zawaideh M, Mossa-Basha M, et al. The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies. Proceedings SPID Medical Imaging 2019. https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10949/2513167/The-utility-of-deep-learning--evaluation-of-a-convolutional/10.1117/12.2513167.short

Monday, February 22, 2021

Diabetes Meets Machine Learning, Part 1

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

Of all the disorders that have responded well to artificial intelligence and machine learning, diabetes mellitus probably tops the list. The evidence supporting a role for Machine Learning (ML)-enhanced algorithms in managing the disease is persuasive and applies to several components of patient care, including screening, diagnosis, treatment, and prognosis.

Let's start with screening: As most clinicians know, there's a difference between screening for disease and diagnosing it. The former casts a much wider net, which means it will include more false positives. Once this larger cohort has been identified, more precise diagnostic testing can be performed to pinpoint patients who have the disorder.  As we explain in our next book, The Digital Reconstruction of Healthcare, screening tools for diabetes have been in existence for many years, including tools to help clinicians and patients identify the presence of prediabetes. Still, new technology has taken the screening process to a new level.

The American Diabetes Association (ADA) defines prediabetes as fasting plasma glucose between 100 and 126 mg/dl, a 2-hour oral glucose tolerance test reading between 140 and 200 mg/dl, or hemoglobin A1c of 5.7% to 6.5%.1 The Association has developed a risk assessment tool to help clinicians and patients; it asks for a patient's age, gender, family history of diabetes, the presence of hypertension, whether the patient is physically active, their ethnicity, and height and weight. Although this kind of scoring system will detect many persons with prediabetes, it has its limitations and ignores many changes in a patient's medical history over time. An artificial intelligence (AI)-enabled assessment tool that takes advantage of gradient boosting, on the other hand, is capable of factoring in many more risk factors gleaned from a patient’s electronic health record. This type of assessment system can accurately predict which patients will progress to full-blown diabetes, with an area under the curve (AUC) of 0.865.2 The gradient boosting approach has proven more accurate than logistic regression and evaluation of standard clinical cutoffs of blood glucose ≥ 110 mg/dl and HbA1c ≥ 6.0%. An analysis by Avivit Cahn from the Hebrew University of Jerusalem and her colleagues found the said ML algorithm incorporated 69 variables to achieve its superior AUC.

There is also reason to believe that AI-enhanced algorithms can help identify individuals before they become prediabetic. The Centers for Disease Control and Prevention (CDC) has a screening test to help predict prediabetes that asks one’s age, family history of diabetes, the existence of hypertension, race or ethnicity, whether one is physically active, gender, height and weight.

But logistic regression, artificial neural networks, random forest analysis, and gradient boosting have uncovered numerous other risk factors to predict prediabetes. Among the predictors that may help refine the risk stratification process: food security, citizenship, monocyte count, uric acid level, ALT, RBC count, and serum calcium and potassium.3 Whereas the CDC screening tool returned an area under the receiver operating characteristic (AUROC) of 64.4% in a large cohort from the National Health and Nutrition Survey, several AI-based algorithms generated AUROCs at or above 70%.

Several research projects suggest machine learning can improve the treatment of diabetes. In our book on mobile medicine, we discussed DreaMed,*  an Israeli vendor that developed DreaMed Diabetes to provide clinical decision assistance to clinicians managing Type 1 diabetes in patients who are using an insulin pump and continuous glucose monitoring (CGM). Its Advisor Pro algorithms collect several data types, including the patient’s daily blood glucose readings, meal carbohydrate data, insulin dosing records, and physical activity level. The software then uses event-driven machine learning and adaptive technology to create a more individualized regimen, adjusting its insulin dosing recommendations and offering behavior-modification recommendations accordingly. DreaMed Diabetes relies on a clinical AI program called MD-Logic. It uses fuzzy logic and adaptive learning algorithms to approximate how diabetes specialists reason as they adjust an individual's insulin regimen. The company claims that the software provides “faster analysis and deeper insights” than would normally be available with routine endocrinologist care. There is research support for this automated approach to Type 1 diabetes management.4

This coordinated interplay between an insulin pump and CGM sometimes referred to as closed-loop control or an artificial pancreas, has also been a focus of Mayo Clinic researchers. Yogish Kudva, MBBS, along with colleagues Sandeep Gupta and Ayan Bannerjee from Arizona State University, has been analyzing continuous glucose monitoring data from people with type 1 diabetes on the Medtronic MiniMed 670G closed-loop system* to develop software that would help each patient and clinic provider improve closed-loop control. They have also developed a system for measuring heart rate, respiratory rate, and self-report symptoms of viral infections in individuals with diabetes. The aim is to investigate the role of glycemic control on the progression of signs and symptoms of infectious disease in both at-home and hospital settings.

Similarly, Dr. Kudva participated in a recent investigation on the role of a closed-loop system to improve glycemic control in adolescents and young adults, part of the International Diabetes Closed-Loop Trial.5

Unfortunately, until recently, CGM has been out of reach for most Type 2 patients. But with the introduction of new, state-of-the-art technology, it is now available to them as well. We’ll explore these innovative solutions in part 2 of our series on machine learning-enhanced diabetes.

*Products and services mentioned are not endorsements.  Mayo Clinic has no relationship with any of these vendors.



1. American Diabetes Association. Classification and diagnosis of diabetes: standards of medical care in diabetes 2019. Diabetes Care. 2019;42(Suppl. 1):S13–S28. doi .org/10.2337/dc19-S002

2.  Cahn A, Shoshan A, Sagiv T el al. Prediction of progression from pre-diabetes to diabetes: development and validation of a machine learning model. Diabetes Metab Res Rev. 2019; e3252. doi.org/10.1002/dmrr.3252

3. De Silva K, Jonsson D, Demmer RT. A combined strategy of feature selection and machine learning to identify predictors of prediabetes. JAMIA. 2020;27: 396–406.

4. Nimri R, Muller I, Atlas E, et al. MD-logic overnight control for 6 weeks of home use in patients with type 1 diabetes: randomized crossover trial. Diabetes Care. 2014;37:3025_32.

5. Isganaitis E, Raghinaru D, Amber-Osborn L et al. Closed-Loop Insulin Therapy Improves Glycemic Control in Adolescents and Young Adults: Outcomes from the International Diabetes Closed-Loop Trial. Diabetes Technol. Ther. 2021l Jan 21. doi: 10.1089/dia.2020.0572.  Online ahead of print.

Friday, February 12, 2021

High-Quality Hospital Care — Minus the Hospital

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

In a time of COVID, the current care models just can’t remain in place. We need innovative ways to address the high cost of acute care. One solution that is taking a front seat is telemedicine.  Telecare has accelerated from 3-4% of visits in January 2020 to 90% in April to a new normal of 20% in 2021. Cultural change has modified patient expectations for the care they can get at a distance, which can be more convenient, less likely to result in COVID exposure, and more patient-centric.

Research has demonstrated that hospital-at-home programs for patients with specific acute medical conditions can reduce complications and reduce the cost of care by 30% or more.[1] One of the most progressive programs to focus on this transition was spearheaded by Johns Hopkins Hospital in 1994. Bruce Leff, MD, and his colleagues have tested this program with 455 elderly patients from three Medicare-managed systems and a VA medical center.[2] They found that the home model met the quality of care standards comparable to those expected of in-hospital programs. Also, "On an intention-to-treat basis, patients treated in hospital-at-home had a shorter length of stay (3.2 vs. 4.9 days) (P =0.004), and there was some evidence that they also had fewer complications. The mean cost was lower for hospital-at-home care than for acute hospital care ($5081 vs. $7480) (P < 0.001).”

A meta-analysis of 61 randomized clinical trials that looked at hospital-at-home projects found that among 42 trials, which included almost 7,000 patients, this approach reduced mortality (odds ratio, 0.81). Similarly, they decreased readmission rates by 25% (odds ratio, 0.75) and lowered costs. The same analysis revealed that treating every 50 patients in such a program saved one life.[3] Realizing the potential advantages of hospital-at-home programs, several large U.S. provider organizations have entered this space in the last few years, including Mayo Clinic, Partners Healthcare/Brigham and Women’s Hospital, and Mount Sinai Health System in New York. Across the globe, there are also significant programs in Australia, South Wales, and Spain.

Advanced Care at Home, a partnership between Mayo Clinic and Medically Home., tracks heart rate, blood pressure, pulse oximetry, temperature, and respiratory rate in its patient population, using Bluetooth-enabled devices wirelessly connected to the Mayo/Medically Home system. It also uses tablets, a back-up battery system and a Wi-Fi phone. There are, however, critical differences between many home-care programs and the Mayo Clinic system. Many hospital-at-home programs are targeted and designed for low-acuity hospital patients. They use physician house calls as the clinical delivery model. They have a short patient engagement period (2-4 days).

The Medically Home affiliated setup is designed to handle an extended length of stay that includes acute, post-acute and preventative care. It uses a scalable “decentralized” model for high-acuity care and can manage a broad set of diverse use cases and support an extensive patient census. The program uses screening, training, contracting, quality management, and technology and converts "post-acute" community-based supply providers into acute-level providers, bringing goods and services to high-acuity patients at home while focusing heavily on the role of paramedics as the centerpiece of its ability to provide rapid-response capabilities. In practical terms, that means paramedics and other providers go into the home while being virtually connected with a centralized medical command center staffed by physicians who guide the care for decentralized patients and the decentralized providers that care for them.

Advanced Care at Home has made measurable progress within a relatively short period, going from a speculative pilot project about a year ago to a business plan that will likely prove profitable in 2022. During a recent Zoom call with Ajani (AJ) Dunn, Administrator for the program, he emphasized, “It’s a story about volume.  As we looked at the model, we asked ourselves: Will it be effective? And we found that by scaling it up to the point where we can take out the fixed cost of a traditional hospital stay and replace that with the small variable costs of each service we deliver in the home, we can have a sustainable program.” Dunn explained that by working through a centralized command center staffed by physicians and getting buy-in from third-party payors, this approach is slowly turning the corner financially. Since most insurers do not have a hospital at home plan built into their policy, that has required the Mayo team to negotiate with payors one by one, explaining the cost-effectiveness of the new model.

The next step in the program’s growth will require finding ways to reinvent the existing system. “The off-the-shelf system we use to administer Advanced Care at Home is well-calibrated for traditional medical admissions, including COPD, CHF, pneumonia and the like. That approach involves drawing labs, titrating medications, etc. But we need other ways to use the system. Our plan now is to embed teams into other disease states, including oncology, cardiovascular disease, and transplantation, to understand the natural progression of each disease and the necessary interventions. Then we can create clinical protocols and administrative logistics that replicate in-hospital care in the home, but customized for each disease state.”

Undoubtedly, concerns about high costs and unexpected complications will continue to dissuade patients from seeking in-hospital care for many years. But as this new model expands, it's likely more patients will see the advantages of seeking high-quality hospital care — minus the hospital.



1. Klein S. “Hospital at home” programs improve outcomes, lower costs but face resistance from providers and payers. The Commonwealth Fund. Accessed January 15, 2020. https://www.commonwealthfund.org/publications/newsletter-article/hospital-home-programs-improve-outcomes-lower-costs-face-resistance

2. Leff B, Burton L, Mader SL, et al. Hospital at home: feasibility and outcomes of a program to provide hospital-level care at home for acutely ill older patients. Ann Intern Med. 2005;143:798–808.

3. Caplan GA, Sulaiman NS, Mangin DA, et al. A meta-analysis of “hospital in the home.” Med J Aust. 2012;197:512–519.

Wednesday, February 10, 2021

Let Your Freak Flag Fly

 John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

If Mary Putnam Jacobi were alive today, she would probably embrace artificial intelligence (AI), machine learning, and data analytics. Dr. Jacobi might best be described as the mother of modern scientific medicine  or at the very least one of its founding parents. In 1868, she was the first woman to enroll in the University of Paris School of Medicine. After graduating in 1871, this unconventional thinker arrived in the U.S., where she advocated for the inclusion of laboratory science, experimentation and statistics as the foundation cornerstone of modern medical practice. Equally important, Jacobi “became a powerful advocate for the equal contribution of women to medicine.” Pushing clinicians to buy into the notion that experimentation and statistics were needed for good quality patient care may seem unimpressive today. Still, it was almost heresy in an age when the received wisdom from one’s medical school professor was all that was necessary to “demonstrate” that a treatment protocol was effective. Given her “color-outside-the-lines” approach, it’s not hard to imagine her becoming passionate about the role of AI in medicine — not as a panacea but as an indispensable adjunct to human reasoning and clinical trials.

The evidence shows that machine learning, advanced data analytics, and subgroup analysis all have the power to reinvent the way patient care is delivered and the potential to personalize that care. But as we said in an earlier blog, while there are several well-documented AI-based algorithms now available, there are also several questionable vendors that have rushed to market with little evidence to support their claims. This dilemma begs the question: How do we ensure that well-supported digital tools get the attention they deserve? Several new guidelines have been published to encourage more robust AI-related research, but clinicians also need to develop machine learning literacy. Put another way, we can all benefit from gaining a better grasp of what’s under the hood, explained in plain, jargon-free English. That was one of the goals of our last book, Reinventing Clinical Decision Support.

Our research demonstrates that clinical decision support certainly needs to be reinvented. If you look back over the last several decades, you’ll find that hundreds of clinical research studies have been misinterpreted. A review of 71 randomized clinical trials in 1987, for instance, concluded that numerous treatments were useless; a closer analysis, however,  found that all 71 had generated false-negative results because the populations they studied were too small — a type 2 statistical error. [1] Fast forward to 1994; a JAMA analysis revealed that 383 RCTs, many published in the world’s top academic journals, likewise jumped to the conclusion that several treatments were ineffective, once again because the trials had enrolled too few patients. [2]

More recently, there’s reason to believe that the bedrock upon which many day-to-day clinical decisions rest is somewhat shaky. It’s a foundation that can be made stronger with ML-based tools like convolutional neural networks, random forest modeling, gradient boosting, and clustering. A case in point: The Look AHEAD study that published in 2013.[3] This RCT assigned over 5,000 overweight patients with type 2 diabetes to either an intensive lifestyle modification program or a control group. The investigators' goal was to determine if the lifestyle program would reduce the incidence of cardiovascular events. The study was terminated early because there were no significant cardiovascular differences between the intervention and control groups.

An ML-fueled analysis that used random forest modeling to re-examine the Look AHEAD data turned these results upside down. During random forest analysis, a series of decision trees are created. Initially, the technique randomly splits all the available data — in this case, the stored characteristics of about 5,000 patients in the Look AHEAD study — into two halves. The first half serves as a training data set to generate hypotheses and construct the decision trees. The second half of the data serves as the testing data set.

Using this technique, Baum et al. constructed a forest that contained 1,000 decision trees. They looked at 84 risk factors that may have been influencing patients’ response or lack of response to the intensive lifestyle modification program, including numerous characteristics that researchers rarely, if ever, consider when doing a subgroup analysis. The random forest modeling also allowed the investigators to examine how these variables interact in multiple combinations to impact clinical outcomes. In the final analysis, Baum et al.[4] discovered that intensive lifestyle modification did prevent cardiovascular events for two subgroups, patients with HbA1c 6.8% or higher (poorly managed diabetes) and patients with well-controlled diabetes (Hba1c < 6.8%) and good self-reported health. That finding applied to 85% of the entire patient population studied. On the other hand, the remaining 15% who had controlled diabetes but poor self-reported general health responded negatively to the lifestyle modification regimen. The negative and positive responders canceled each other out in the original Look AHEAD statistical analysis, falsely concluding that lifestyle modification was useless.

AI-enhanced data analyses like this only serve to reinforce Maru Putnam Jacobi’s contention that coloring outside the lines will propel health care in new directions. Her unconventional mindset, and ours, are best summed up in the words of Crosby Still and Nash: “Let your freak flag fly!”


1. Freiman JA et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. N Engl J Med. 1978; 299:690-694.

2Moher D et al. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA. 1994; 272:122.

3. Look AHEAD Research Group. Cardiovascular effects of intensive lifestyle intervention in type 2 diabetes. N Engl J Med. 2013. 369:145-154.

4. Baum, A et al. Targeting Weight Loss Interventions to Reduce Cardiovascular Complications of Type 2 Diabetes: A Machine Learning-Based Post-Hoc Analysis of Heterogeneous Treatment Effects in the Look AHEAD Trial. Lancet Diabetes Endocrinology, 2017;  5: 808–815.

Thursday, February 4, 2021

The Therapeutic Potential of Voice Technology

John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform, wrote this article.

The human voice is capable of extraordinary feats of genius and everyday acts of kindness.  It can recite Shakespearean sonnets, teach our children moral values, stir audiences with a dramatic performance, and much more. But few of us ever imagined it capable of assisting in the diagnosis of disease. That’s about the change, as evidenced by several innovative projects in voice technology.

With the right digital tools, it is now possible to analyze a patient’s speech patterns to detect clues to underlying pathological issues. Elad Maor, MD, Ph.D., with the Mayo Clinic Department of Cardiovascular Medicine, and his colleagues have looked at voice samples from about 100 patients who underwent coronary angiograms, asking them to read text excerpts and respond to questions about positive and negative emotional experiences. Their recorded responses found subtle differences in vocal pitch and intensity between patients who were ultimately diagnosed with heart disease and normal controls. [1] Dr. Maor and his associates concluded: “One possible explanation for our interesting finding is the documented association between mental stress, the adrenergic system, and voice. . . . Emotional stress conditions change the human voice, including an increase in fundamental frequency. . . . [O]ne possible hypothesis to interpret our findings is that the association between voice and atherosclerosis is mediated by hypersensitivity of the adrenergic system to stress. The association between stress, the adrenergic system, and atherosclerosis is well established on the basis of robust data.”

There is also reason to believe that voice technology may help detect pulmonary hypertension. Jaskanwal Deep Singh Sara, M.B., Ch.B. (also with the Mayo Clinic), in collaboration with scientists from Vocalis Health (an Israeli vendor), has analyzed voice recordings among patients who had invasive cardiac hemodynamic testing, the standard approach to diagnosing pulmonary hypertension. The recordings were analyzed to measure pitch, loudness, jitter, and other metrics. Dr. Sara and his colleagues found a significant association between an invasively derived hemodynamic index used to measure pulmonary hypertension and the vocal biomarkers. Patients with pulmonary arterial pressure at or above 35 mmHg had higher mean vocal biomarker readings than those with pressure readings at or below 35 mmHg. Given that invasive testing occurred during cardiac catheterization, the non-invasive collection of a patient's voice patterns holds promise as a safer alternative. If controlled clinical trials confirm the findings, it will likely reduce the cost and risk associated with pulmonary hypertension (PH) diagnosis. [2]

Investigators in Belgium also have had success using vocal characteristics as part of a suite of tools to help detect Parkinson's Disease in its early stages. [3]  Some studies suggest that 60-90% of patients have subtle changes in their voice and speech patterns when initially diagnosed. In situations in which the diagnosis is questionable, however, a neurologist might administer a loading dose of levodopa, one of the standard drugs used to treat the disease and watch for improvements in a patient’s speech and voice. After administering the drug, the Flemish researchers monitored the strength of a patient's facial or mouth muscles and evaluated vocal quality, frequency, breathiness, phonation time — how long a person can sustain a vocal sound on one deep breath — and several other parameters, folding them into a metric called a Voice Handicap Index. They found that these markers helped distinguish patients with idiopathic Parkinson's disease from healthy individuals.

Mayo Clinic is also exploring the value of voice technology, coupled with artificial intelligence, to address patients' needs with neurological disorders and related motor speech disorders. The project's goals include creating digital tools for voice-based disease detection in the office, over the phone, and in a patient's home; earlier, more accurate and more holistic diagnoses; and the provision of individualized, in-home markers of disease progression and treatment response. To accomplish those goals, the Neurology AI Program and other Mayo researchers and clinicians are developing a fully automated digital speech diagnostics platform that can take a patient's speech sample and provide probabilistic diagnoses. Those taking the lead in this initiative believe that the primary goal should not be to map directly onto a clinical diagnostic label but instead use AI to extract clinically meaningful information from a speech sample. 

Mayo Clinic-based speech pathologists Darley, Aronson and Brown [4] proposed a similar idea in 1969 in their seminal work on the dysarthrias, speech disorders caused by muscle weakness.  While this theoretical construct was previously based on conventional speech labels, advances in AI have made it feasible to build such feature space using latent patterns in the data, which can then be labeled and linked to dense, multivariate medical data. This Deeply Annotated Speech Latent space can then be used to characterize new speech samples. When connected to non-speech medical and demographic data, a broader neurologic or systemic diagnosis can be rendered. The model will likely have the most considerable impact if deployed on smartphone devices, smart speakers or other wearable technology.

Voice technology has yet to surpass Shakespeare’s genius, replace a gifted actor's heart-wrenching performance,  or serve as a moral compass for the next generation. Still, the evidence suggests it is ushering in a new generation of intelligent digital tools that will likely transform patient care. 


1. Maor E, Sara JD, Orbelo DM, et al. Voice signal characteristics are independently associated with coronary artery disease. Mayo Clinic Proceedings. 2018;93:840–847.

2. Sara JDS, Maor E, Borlaug B, et al. Non-invasive vocal biomarker is associated with pulmonary hypertension. PLOS ONE. 15(4):e0231441. doi.org/10.1371/journal.pone.0231441

3. Lechien JR, Delsaut B, Abderrakib A et al. Orofacial Strength and Voice Quality as Outcome of Levodopa Challenge Test in Parkinson Disease. Laryngoscope, 130:E896–E903, 2020.

4. Frederic L Darley, Arnold E Aronson, and Joe R Brown. Clusters of deviant speech dimensions in the dysarthrias. Journal of speech and hearing research, 12(3):462–496, 1969.

Tuesday, January 26, 2021

Wearable Danger

This article is written by John Halamka, M.D., president, Mayo Clinic Platform, and Paul Cerrato, senior research analyst and communications specialist, Mayo Clinic Platform.

If you ask health care executives what keeps them up at night, many would sum up their worries in one word: ransomware.  By one estimate, 56% of organizations suffered a ransomware attack in the last year. While there are countless ways in which a cyberthief can penetrate a facility’s computer network to block access to essential data, one avenue that gets too little attention is through wearables and related medical devices.  A growing number of providers are now allowing patients to send data from blood glucose monitors, blood pressure cuffs, bed sensors, and portal EKG devices to their networks. And during the COVID-19 pandemic, many more clinicians are working remotely using their own laptops, tablets and smartphones to access a hospital or office EHR system. All these connections are potential opportunities for hackers to infiltrate your computer network. And the word potential doesn’t fully capture the danger.

In 2019, for instance, FDA issued an alert to health professionals warning about a cybersecurity vulnerability affecting Medtronic implantable cardiac devices (ICDs), programmers and home monitors. The agency found the vulnerability in the wireless telemetry technology used to communicate between the ICDs, clinic programmers and home monitors. Similarly, the company that makes the OneTouch insulin pumps contacted patients using the device of the possibility that it could be hacked and reprogrammed, which could have life-threatening consequences.

During a recent conversation with Leon Lerman, CEO of Cynerio, a cybersecurity solutions firm, he explained that once a hacker infiltrates a computer network, often through a phishing scam and malware, medical devices become easy targets. That’s the case for several reasons, including inadequate segmentation and outdated operating system. Virtual local access networks (VLANs) are one way to address the issue because they limit the number of users allowed to have access to a specific part of the network.  Unfortunately, a study sponsored by Forescout, a security firm, found “only 49 percent of medical devices were deployed across 10 virtual local access networks (VLANs) or fewer ….”

Outdated operating systems, an easy access point for hackers, remain a persistent problem for health care providers, as are outdated software applications. An international survey involving 600 health care IT professionals in 2019 found more than 1 out of 4 organizations were still running Windows 7 on their medical devices. The danger posed by this practice may not be immediately obvious to most clinicians, but because many older OSs are no longer supported by their manufacturers, security patches are no longer available to block newly designed digital threats. Of course, health care providers running currently supported operating systems can also fall victim to cyberattacks if they fail to install security updates as soon as they become available. That’s how the infamous WannaCry ransomware worm was able to penetrate the NHS and numerous other networks; it affected more than 200,000 computers worldwide in 150 countries. Microsoft had already issued a security patch before the WannaCry incident, but many organizations had neglected to install it in time.

One of the challenges in keeping operating systems up to date is the restrictions that hospital IT teams face when they try to address the issue. Most devices are black boxes in the sense that the manufacturer does not allow users to touch the software; doing so without the company’s permission usually voids the warranty. That makes it virtually impossible for a hospital or medical practice to install security updates to legacy OSs, even when they are available. If the device manufacturer is cooperative, it may be possible to have their technicians do these updates. When that’s not an option, segmentation becomes all the more important.

Fortunately, many device manufacturers are now beginning to realize that their reputations depend upon developing machinery that is not just clinically functional but hardened to cyberattacks. Many new devices come with a Manufacturer Disclosure Statement for Medical Device Security (MDS2) that spells out the security protocols used on the device, whether anti-malware software has been installed, and whether it should even be connected to the Internet.

The adage about necessity being the mother of invention certainly applies to the Internet of Medical Things. As the health care ecosystem experiences more cyberattacks, we learn to adapt, and out of necessity, develop creative tools to defend our networks and most importantly we learn more effective ways to protect our patients—our number one priority.

Friday, January 22, 2021

How is AI Impacting Health Care Today?

By John Halamka and Paul Cerrato*

We are often asked this question during interviews, podcasts and speaking engagements. It’s a complicated question that requires context. A closer look at the research and product offerings in digital health demonstrates that there are several high-quality, well-documented algorithms now available, but there are also several questionable vendors that have rushed to market with little evidence to support their claims. Separating the wheat from the chaff can be a full-time occupation.

We recently received a press release from a large U.S. company highlighting a new AI system that may be able to diagnose dementia using a patient’s vocal patterns. The vendor pointed out that its research generated an area under the curve (AUC) of 0.74 for its system, which suggests that at least one in 4 patients with dementia would be overlooked. With these concerns in mind, the question is: What kind of guidelines can clinicians and technologists on the front lines turn to when they want to make informed choices?

Ideally, we need an impartial referee that can act as a Consumer Reports type service, weighing the strengths and weaknesses of AI offerings, with brief summaries of the evidence upon which they base its conclusions. In lieu of that, there are criteria that stakeholders can tap to help with the decision making process. As we point out in a recent review in the New England Journal of Medicine journal, called NEJM Catalyst, at the very least, there should be prospective studies to support any diagnostic or therapeutic claims. Too many algorithms continue to rely on retrospective analysis to support their products. (1) In our NEJM analysis, we include an appendix entitled “Randomized Controlled Trials and Prospective Studies on AI and Machine Learning,” which lists only 5 randomized controlled trials and 9 prospective, non-RCT studies. When one compares that to the thousands of AI services and products coming to market, it’s obvious that digital health still has a long journey to make before it’s fully validated.

That’s not to suggest that there are no useful, innovative AI and machine learning tools that are well supported, as well as several that are coming through the pipeline.  There are credible digital tools to estimate a patient’s risk of colorectal cancer (ColonFlag), manage type 1 diabetes (DreaMD), and screen for diabetic retinopathy (IDx), all of which are supported by good evidence.** The FDA has also published a database of approved AI/ML-based medical technologies, summarized by Stan Benjamens and his associates in npj Digital Medicine.(2) (Keep in mind when reviewing this database, however, that some of the algorithms cleared by FDA were based on very small numbers of patients.)

A recent virtual summit gathered several thought leaders in AI, digital health and clinical decision support to create a list of principles by which such tools can be judged. Spearheaded by Roche and Galen /Atlantica, a management consulting firm, its summit communique refers to the project as “A multi-stakeholder initiative to advance non-regulatory approaches to CDS quality.”  Emphasizing the need for better evidence, the communique states: “The development of CDS is driven by increasing access to electronic health care data and advancing analytical capabilities, including artificial intelligence and machine learning (AI/ML). Measures to ensure the quality of CDS systems, and that high-quality CDS can be shared across users, have not kept pace. This has led some corners of the market for CDS to be characterized by uneven quality, a situation participant likened to “the Wild West.”

The thought leaders who gathered for the CDS summit certainly aren’t the only ones interested in improving the quality of AI/ML-enhanced algorithms.  The SPIRIT AI and CONSORT-Ai Initiative, an international collaborative group that aims to improve the way AI-related research is conducted and reported in the medical literature, has published 2 sets of guidelines to address the issues we mentioned above. The twin guidelines have been published by Nature Medicine, BMJ and Lancet Digital Health. (3,4) They are also available on the group’s web site.

With all these thought leaders and experts on board, there’s no doubt the AI ecosystem is gradually transitioning from the “Wild West” into a set of well-defined and repeatable processes that health care stakeholders can trust. http://geekdoctor.blogspot.com/2021/01/to-build-fire.html


*Paul Cerrato is a senior research analyst and communications specialist at Mayo Clinic Platform

 **Products mentioned are not endorsements.  Mayo Clinic has no relationship with any of these vendors.



1. Halamka J, Cerrato, P. The Digital Reconstruction of Health Care. NEJM Catalyst: Innovations in Care Delivery. Vol 1 (6); Nov-Dec 2020.

2. Benjamens S, Dhunnoo P, Mesko B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. Npj Digital Medicine. 3:118, 2020. https://www.nature.com/articles/s41746-020-00324-0

3. Cruz Rivera, S, Liu, X, Chan, A-W et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nature Medicine volume 26pages13511363(2020)

4. Liu, X, Cruz Rivera, S, Moher D et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension Nature Medicine volume 26pages13641374(2020)