2009 will be a challenging year as budgets are constrained, a new administration is launched, and more demands are placed on IT to help stakeholders work more efficiently.
Here are my goals for 2009.
BIDMC
1. I will continue to evolve cloud computing offerings in 2009, including the rollout of a new storage design, offering generous amounts of low cost space for clinical care, email, and research.
2. I will continue the rollouts of software as a service EHRs for our community providers including connectivity to the MAeHC quality warehouse, the MA-Share healthcare information exchange, and Google Health/Microsoft Healthvault.
3. I will work collaboratively with all our corporate communications, clinical, and administrative stakeholders to launch a new intranet that includes many social networking features.
4. I will support the Federal and State economic stimulus activities for Healthcare IT.
HMS
1. I will work with Public Affairs, research, and administrative leaders to redesign and unify the external web presence at Harvard Medical School, which today has many disparate websites with heterogeneous navigation and search.
2. I will work with administrative stakeholders to implement a next generation of web-based workflow tools which improve efficiency by automating document management, routing, and communication.
3. As with BIDMC, I will evolve my cloud computing offerings for storage (high performance, mid-tier, archive) and implement the first pilots of our New England Biocomputing Center.
4. I will support the Federal and State economic stimulus activities for Healthcare IT.
MA-Share/NEHEN
I've listed these two organizations together because my goal is to ensure a successful merger of these two complementary organizations in 2009, creating a single statewide health information exchange for all types of data among all stakeholders. The combined organization will work with all the other great ehealth organizations in Massachusetts to support the Federal and State economic stimulus activities for Healthcare IT.
HITSP
By charter, which limits me to two terms, I must hand off the HITSP chairmanship on October 1, 2009. What will I focus on for these next 10 months?
1. I will work with all stakeholders to ensure we have all the standards we need for exchange of clinical summaries, both simple and complex, that meet all the data exchange requirements of EHRs and PHRs.
2. I will work with all stakeholders to move HITSP interoperability specifications to a service oriented architecture, making implementation much easier, such as conforming to a published WSDL
3. I will work with HITSP teams to develop a data dictionary for interoperability, making it easy for an implementer to locate all the standards they need for specific data elements. I will work with all the CEOs of EHR vendors and CCHIT to ensure we reduce their barriers to standards implementation.
4. I will support the Federal and State economic stimulus activities for Healthcare IT.
Personal
My personal goals will stay very focused in 2009
1. Continue my vegan lifestyle, eating locally grown foods
2. Continue my outdoor activities - hiking/running, climbing, kayaking, cycling, skiing
3. Continue my Japanese flute (Shakuhachi) and Turkish Flute (Ney) training
4. Continue my "Dreams of Green", learning more about systems to heat/cool and generate power using sustainable, non-polluting technologies.
I'm sure 2009 will have its economic bumps, but the future looks very bright.
Wednesday, December 31, 2008
Tuesday, December 30, 2008
Looking back on 2008
It's the eve of New Year's Eve. Since I'll be posting my 2009 New Year's resolutions tomorrow, today's blog is a look back at my 2008 resolutions. How did I do and most importantly what did I learn along the way? This is a very long blog entry, so my apologies to readers on iPhones and Blackberrys.BIDMC
1. One of my major 2008 goals for BIDMC was to build a cloud computing center hosting EHRs for community clinicians in Massachusetts via software as a service of approach.
Status - the center is live and 4 practices are fully implemented. We're implementing 6 practices per month throughout 2009 and 2010. We're in progress creating a connection the the Mass eHealth Collaborative quality data warehouse, integrating with the MA-Share health information exchange, and piloting Google Health PHR integration.
Lessons learned - Free is not cheap enough for primary care physicians in the US. Stark safe harbors enable hospitals to subsidize 85% of the costs of EHR implementation. Many PCPs do not have enough spare cash to fund the 15% plus office hardware. In 2009, I'll work with our physicians' organization (not bound by Stark funding restrictions) to develop a strategy to reduce the cost even further.
2. BIDMC has centralized just about all IT application and infrastructure services. My 2008 goal involved creating a plan for centralizing image lifecycle management of radiology, cardiology, pulmonary, GI, and Ob/Gyn.
Status - We've made progress but still have work to do. We've created the governance to do this - an Enterprise Image Management Steering Committee including all departmental stakeholders. We've assigned project management responsibility. Enterprise Image management software selection is led by a consultant, Joe Marion, one of the most knowledgable PACS experts in the country. Infrastructure design and PACS disaster recovery efforts are led by my internal management and staff.
Lessons learned - In the image management industry, products are still rapidly evolving. We're approaching this project using hardware (EMC Atmos) and software that did not exist when I wrote my 2008 goal. We plan to pilot of our design in 2009 by unifying all cardiology image sources (cath, echo, vascular).
3. 2008 was a tumultuous time for my infrastructure teams since we moved over 2000 people for the opening of our new Center for Life Sciences research building. Completing this project plus those of our physician practices, community sites, and our internal moves led to many deferred vacations, long nights and weekend work. One of my goals for 2008 was to ensure the moves/adds/changes plans were better communicated from business owners to IS.
Status - We've increased the size of our project management and community support teams, built a standardized process for project intake, and assigned an IT liaison to our facilities and community practice departments.
Lessons learned - There is no substitute for formal project management. Every project requires communication, coordination, and resource allocation. Whenever I'm asked to bypass my usual processes to accelerate projects, the answer is that workarounds will actually slow down the project.
Harvard Medical School
1. The Clinical Translational Science Awards (CTSA) brought over $200 million in NIH funding and in kind contributions to Harvard Medical School. One of my goals for 2008 was to build the social networking infrastructure to support collaboration throughout the Harvard Empire.
Status - Catalyst is live and includes a very powerful function called Profiles to identify collaborators based on the their publications and funding. In 2009, Profiles will include even more advanced tools for active and passive networking
Lessons Learned - Every aspect of catalyst is via Google Analytics so we know what is used and how. Our plans for features and new applications were markedly changed to match the demands of our users.
2. In an era of budget reductions (Harvard's endowment losses have led the medical school to implement a 10% cut in all departments), strong financial management is a necessity. Once of my goals was to implement financial analytics/business intelligence and not just a budgeting function in IS.
Status - we hired a business manager who brings thoughtful analysis to IS financial operations.
Lessons Learned - As organizations mature, their needs change. The budgeting function of the past needed to evolve into strong financial analysis for us to manage the budget as tightly as the economy demands.
3. When budgets are tight, economies of scale must be exercised whenever possible. In 2008, one of my goals was enhance our centralized high performance computing capabilities.
Status - During our 2008 Harvard High Performance Computing Symposium, we announced the New England Biocomputing Center, a collaborative of many academics, researchers, and industrial affiliates that will serve as a resource for all of New England. Thus far, we've had substantial commitments to donate and participate in building our economies of scale beyond just one school.
Lessons Learned - Cloud computing for research offers such great capabilities and low cost that political barriers are not an issue.
MA-SHARE
1. In 2008, we continued the expansion of our statewide e-prescribing gateway and our health information exchange of clinical summaries. My goal was to increase the number of organizations exchanging data.
Status - e-prescribing is live for all the physicians of BIDMC and Partners healthcare. Clinical data exchange is live for BIDMC and its network of private clinicians. Children's and Northeast Health System are live. Others will soon follow.
Lessons learned - Health information exchange for technology's sake is not sustainable. There must be a business value case. In Massachusetts, the exchange of medication transactions and sharing of clinical summaries brings cost avoidance that payers and providers are willing to fund.
NEHEN
1. NEHEN has been the administrative healthcare information exchange for Massachusetts since 1997. Our implementations started with institutions and large provider organizations than expanded to community hospitals. My 2008 goal was to expand NEHEN capabilities to small providers.
Status - We have made significant progress in 2008, approaching our goal of adding an additional 80 organizations to the data exchange.
Lessons Learned - As with our EHR implementation, we learned that physicians in small practices are struggling. We reduced the price for solo practitioners to accelerate adoption.
HITSP
1. AHIC gave HITSP 3 use cases in 2006, 4 in 2007 and 6 in 2008. The 2008 Use Cases were particularly difficult because of rapid change in some of the domains, such as the genome and consumer healthcare devices. My goal was to facilitate standards harmonization for the genome/family history, secure provider/patient messaging, connectivity to healthcare devices, public health reporting, immunization history, and consultation/transfers of care.
Status - all HITSP deliverables were completed and presented to AHIC.
Lessons Learned - Implementation is a very important measure of standards harmonization success. HITSP work products are implemented by several organizations now with more to follow in 2009. The Value Case approach suggested by the AHIC Successor prioritizes work for HITSP and CCHIT based on factors such as customer demand and likelihood of implementation will accelerate interoperability in the US in 2009.
Personal Goals
1. My 2008 personal goals included a major effort to further refine my vegan lifestyle to eating locally and regionally. From May until November we grew our own vegetables and picked up weekly vegetable shares from our CSA, Redfire Farm. We canned throughout October and now continue to enjoy vegetables from canning stores and root cellar.
2. I tried very hard to fly less in 2008, using Webex, teleconferences, and trains as much as possible. I still traveled too much. Cisco has agreed to partner with me on a home telepresence pilot. I'll blog about that effort next month. If works well, I may be able to reduce many flights in 2009.
3. In 2008, I've kept up my physical activity with kayaking from April through October, cycling May through November, climbing June through October, skiing December through March and hiking year round. I did climb the Eichorn pinnacle (photo above) and played my Japanese flutes on many summits. My daughter joined cross country and track so our walks have turned into runs together.
Status - the balance of family, personal and work time seems to be working
Lessons learned - a vegan lifestyle, consistent exercise, and reserved time for family have resulted in a happy and healthy 2008.
Tomorrow - a glimpse of what I hope 2009 will bring.
Monday, December 29, 2008
The Broken Window Effect
As an adult I've returned to various locations from my childhood and found the white picket fences, station wagons, and neighborhood shops transformed into rough, run down, and unsafe neighborhoods. This did not happen overnight. What happened in these places is the same thing that can happen in a business or your personal life. I call it the "Broken Window Effect"
Imagine the perfect "Lake Wobegone" neighborhood where everything is above average. A baseball goes through a window, but the owner decides not to fix it. Then, because the house looks a bit shabby, another neighbor leaves a junked car on the street. Then a bit of graffiti is not cleaned up. Then folks stop picking up garbage from their yards.
The can happen inside a house. One pile on the floor doesn't take too much room, so a few more piles are put around it. Before long, all floor spaces have piles on them. Maintenance items are deferred and junk is not tossed. Years pass and eventually the house is unhealthy to live in, but no one really notices because it happened so gradually.
In IT organizations the Broken Window Effect can occur when management begins to tolerate downtime, constant workarounds, and broken processes.
How do we prevent the Broken Window Effect?
Every downtime incident is investigated within hours of the problem, and a full report is issued to our weekly change control board meeting. The meeting is not punitive, it is a learning environment attended by all my technical managers so that the entire organization can learn together. Questions include
Was there a process failure?
Was there a training failure?
Was there a policy failure?
Was there a planning failure?
Was there a lifecycle maintenance failure?
By examining ever incident when it happens and by building a culture that encourages constant improvement based on collective sharing of our experiences, we ensure that "broken windows" are fixed and that problem recurrence is minimized.
The change control board was created after my Network collapse in 2002 because at that time we discovered several aspects of the IS organization that needed improvement such as
Lack of transparency to downtime with details not openly shared among all groups
Silos of technical knowledge
A tendency to work around and patch rather than identify and correct root causes of problems
A lack of planning projects as a coordinated whole with all services - applications, networks, servers, storage, desktop - considered components of a single comprehensive implementation.
The change control board is so rigorous that even I can get into trouble. I recently implemented a health information exchange application update and did not discuss it at the change control board. Thinking that it was just a minor update, I assumed that there were no infrastructure implications. However, given the fact that the application exchanges data securely outside our firewall, involves databases, integration engines, and application teams, it was important to brief everyone first. My next directors meeting will include an overview of all our health information exchange projects - past, present and future - for all IS stakeholders.
On a personal level, I also try to avoid the broken window effect by renewing/maintaining all aspects of my life ie.
I erase all emails older than 90 days and all files older than 1 year. It's really rare that issue has not been resolved after 90 days or someone requests a file older than a year.
I replace my laptop every 2 years
I replace my blackberry every 2 years
I replace my clothes every 3 years
I keep no paper of any kind in my office and very little in my home. All my reading materials are digital.
Every season has its activities that lead to renewal - Spring house cleaning, Summer planting, Fall yard cleanup, Winterization to prep the house for cold weather.
Whether it's your neighborhood, your home or your office, I recommend you stay vigilant for the Broken Window Effect. Fixing all those broken windows keeps everyone engaged in renewal.
Imagine the perfect "Lake Wobegone" neighborhood where everything is above average. A baseball goes through a window, but the owner decides not to fix it. Then, because the house looks a bit shabby, another neighbor leaves a junked car on the street. Then a bit of graffiti is not cleaned up. Then folks stop picking up garbage from their yards.
The can happen inside a house. One pile on the floor doesn't take too much room, so a few more piles are put around it. Before long, all floor spaces have piles on them. Maintenance items are deferred and junk is not tossed. Years pass and eventually the house is unhealthy to live in, but no one really notices because it happened so gradually.
In IT organizations the Broken Window Effect can occur when management begins to tolerate downtime, constant workarounds, and broken processes.
How do we prevent the Broken Window Effect?
Every downtime incident is investigated within hours of the problem, and a full report is issued to our weekly change control board meeting. The meeting is not punitive, it is a learning environment attended by all my technical managers so that the entire organization can learn together. Questions include
Was there a process failure?
Was there a training failure?
Was there a policy failure?
Was there a planning failure?
Was there a lifecycle maintenance failure?
By examining ever incident when it happens and by building a culture that encourages constant improvement based on collective sharing of our experiences, we ensure that "broken windows" are fixed and that problem recurrence is minimized.
The change control board was created after my Network collapse in 2002 because at that time we discovered several aspects of the IS organization that needed improvement such as
Lack of transparency to downtime with details not openly shared among all groups
Silos of technical knowledge
A tendency to work around and patch rather than identify and correct root causes of problems
A lack of planning projects as a coordinated whole with all services - applications, networks, servers, storage, desktop - considered components of a single comprehensive implementation.
The change control board is so rigorous that even I can get into trouble. I recently implemented a health information exchange application update and did not discuss it at the change control board. Thinking that it was just a minor update, I assumed that there were no infrastructure implications. However, given the fact that the application exchanges data securely outside our firewall, involves databases, integration engines, and application teams, it was important to brief everyone first. My next directors meeting will include an overview of all our health information exchange projects - past, present and future - for all IS stakeholders.
On a personal level, I also try to avoid the broken window effect by renewing/maintaining all aspects of my life ie.
I erase all emails older than 90 days and all files older than 1 year. It's really rare that issue has not been resolved after 90 days or someone requests a file older than a year.
I replace my laptop every 2 years
I replace my blackberry every 2 years
I replace my clothes every 3 years
I keep no paper of any kind in my office and very little in my home. All my reading materials are digital.
Every season has its activities that lead to renewal - Spring house cleaning, Summer planting, Fall yard cleanup, Winterization to prep the house for cold weather.
Whether it's your neighborhood, your home or your office, I recommend you stay vigilant for the Broken Window Effect. Fixing all those broken windows keeps everyone engaged in renewal.
Friday, December 26, 2008
Cool Technology of the Week
Admittedly, this cool technology of the week entry is about an idea, since I have not used the product/service directly.
As I've written about many times, I'm passionate about protecting patient privacy.
The new Massachusetts Data Protection regulations require reporting of breaches and mandate compliance with several new security practices. Many hospitals outsource various aspects of their medical records workflow such as transcription, optical scanning, and data entry of scanned forms. When these services are purchased offshore, how can we reduce the risk that personally identified data will be compromised or identity stolen?
The folks at Tech Speed have a creative answer called Imagefracture.
They provide an editor to markup electronic images and identify fields containing sensitive data. Information that might be used in combination to compromise privacy can then be separated into separate scanned images.
Each of these images is sent to a different work team but enough metadata is retained centrally to reassemble the parts. Here's an example of how it works - suppose that form is comprised of name, address, social security number, medications, and problem lists. If the name is sent to one work group, the address to another, the social security number to another, and the medications/problems lists to another, no single work group has enough information to breech privacy or steal identity.
Thus, offshore data entry or business process outsourcing can continue without fear of data compromise. It has the added benefit that individual outsourced work groups do not need security policy audits for HIPAA, PCI, or SOX requirements.
There have been case reports of rogue employees at outsourcing operations threatening clients with data disclosures and blackmailing them to prevent these releases. Since some countries do not have strict privacy laws, there may not be criminal penalties for inappropriate data releases.
Breaking up work into non-identifiable pieces to protect privacy via a customized image editing and metadata management system. That's cool!
As I've written about many times, I'm passionate about protecting patient privacy.
The new Massachusetts Data Protection regulations require reporting of breaches and mandate compliance with several new security practices. Many hospitals outsource various aspects of their medical records workflow such as transcription, optical scanning, and data entry of scanned forms. When these services are purchased offshore, how can we reduce the risk that personally identified data will be compromised or identity stolen?
The folks at Tech Speed have a creative answer called Imagefracture.
They provide an editor to markup electronic images and identify fields containing sensitive data. Information that might be used in combination to compromise privacy can then be separated into separate scanned images.
Each of these images is sent to a different work team but enough metadata is retained centrally to reassemble the parts. Here's an example of how it works - suppose that form is comprised of name, address, social security number, medications, and problem lists. If the name is sent to one work group, the address to another, the social security number to another, and the medications/problems lists to another, no single work group has enough information to breech privacy or steal identity.
Thus, offshore data entry or business process outsourcing can continue without fear of data compromise. It has the added benefit that individual outsourced work groups do not need security policy audits for HIPAA, PCI, or SOX requirements.
There have been case reports of rogue employees at outsourcing operations threatening clients with data disclosures and blackmailing them to prevent these releases. Since some countries do not have strict privacy laws, there may not be criminal penalties for inappropriate data releases.
Breaking up work into non-identifiable pieces to protect privacy via a customized image editing and metadata management system. That's cool!
Wednesday, December 24, 2008
My Favorite Vegan Resources
Since I'll be taking tomorrow off to spend Christmas with my family, I'm writing my typical Thursday more personal blog entry today. I'm often asked about resources for adopting a vegan diet and lifestyle. Here are my favorites:
Favorite Vegan Web Resources
The Vegetarian Resource Group
Bryanna's Vegan Feast
Fat Free Vegan Kitchen
Vegan Food
Cool Vegan
Happy Cow
Favorite Books
Becoming Vegan: The Complete Guide to Adopting a Healthy Plant-Based Diet
Cooking the Whole Foods Way
This Crazy Vegan Life
Vegan with a Vengeance
Veganomicon
Vegan Express
The Joy of Vegan Baking
The China Study
The Food Revolution
The Gluten Free Vegan
I'm always happy to answer questions and offer advice. After more than 5 years as a Vegan, I can say that it is not a diet, it's my lifestyle. I have no regrets, no cravings, and no desire to return to my old carnivore ways.
Favorite Vegan Web Resources
The Vegetarian Resource Group
Bryanna's Vegan Feast
Fat Free Vegan Kitchen
Vegan Food
Cool Vegan
Happy Cow
Favorite Books
Becoming Vegan: The Complete Guide to Adopting a Healthy Plant-Based Diet
Cooking the Whole Foods Way
This Crazy Vegan Life
Vegan with a Vengeance
Veganomicon
Vegan Express
The Joy of Vegan Baking
The China Study
The Food Revolution
The Gluten Free Vegan
I'm always happy to answer questions and offer advice. After more than 5 years as a Vegan, I can say that it is not a diet, it's my lifestyle. I have no regrets, no cravings, and no desire to return to my old carnivore ways.
Tuesday, December 23, 2008
Troubleshooting Complex IT problems
Whenever I'm asked to solve an intermittent IT problem such as a occasional slowness, occasional lost data or intermittently failing hardware, the research is often complex.
Here's a brief example of the efforts we employ to solve IT mysteries.
A few weeks ago we were told that the Interpreters at BIDMC often received pages with ambiguous call back numbers. At BIDMC, valid numbers are 5 digit extensions, 7 digit local numbers and 10 digit long distance numbers. Interpreters often received 4 digit or 6 digit numbers that were impossible to call back.
The most obvious explanation for such an intermittent problem that only seemed to occur in one department was human error. Doctors misdialed numbers, assuming the last 4 digits would be enough to identify their call back number.
We sent out a broadcast email instructing the clinicians to always dial at least 5 digits.
That did not cure the problem.
We then began a data analysis. Could we relate the bad pages to a particular individual, department or location. We found no correlation.
We then asked if the problem was truly isolated to Interpreters. Our data analysis suggested that it occurred regularly in several departments. No others had mentioned it, but the problem was real.
We then asked if the problem was unique to BIDMC, since we share a paging system with other hospitals. The analysis suggested that it was unique to us, since other hospitals did not have the problem.
It seemed unlikely that just our doctors were using the paging system improperly, so we began analyzing all the hardware involved in paging - phones, interface boards, servers and software. Since some of these components were redundant we experimented with taking one member of clustered services offline to see if we could isolate a problem in one switch, one signal processor or one server. Still no resolution.
We then spoke with the manufacturer of the paging software. They had no reports of similar problems. When then spoke to engineer who wrote the software. He had an idea.
When you use a paging system, the dialog goes something like this
"Please dial the person you wish to page"
User enters pager number
"Please enter your call back number"
User enters call back number
"Thank you"
Page is sent
Many interactive telephone systems have a "buffer clearing function" that clears any input before users enter numbers to eliminate digits being carried over between voice prompts and to cancel out any background sounds that might have occurred between prompts. This sounds great, but what happens if an experienced user begins to dial immediately without listening for the prompts i.e. a person dialing very fast, just immediately enters the pager number, then pauses for 1 second before the "Please enter your callback number" prompt is completed, and enters the call back number. It's likely call back number digits would be truncated.
We turned off buffer clearing and here's what happened.
------Percent Bad Call Back Numbers------
Pager Pre-change Post-change
Russian interpreter 163/493 = 33% 3/244 = 1%
Spanish interpreter 262/959 = 27% 4/404 = 1%
Chinese interpreter 207/1086 = 19% 8/417 = 2%
Of the 15 pages in the post-change group listed as bad, most, if not all, were likely miss-dialed. For example, a couple were "123". Others were bad, but were immediately followed by what appeared to be a correct return page suggesting the caller knew they had entered bad data.
The troubleshooting process was complicated by the fact that truncation only occurred with pagers signed out to other numbers as is the case with Interpreter Services. The "clear buffer" had no impact on pagers not signed out. We tried many times to "type ahead" call back numbers for these and were unable to mimic the problem. It appears to have only been a problem if someone signed their pager out to another. It turns out that the buffer clearing software works differently for pagers with a status of "signed out/covered by"
My advice for diagnosing complex operational IT problems is to work stepwise with every layer of architecture, and isolate the problem. Then, follow the advice of Sherlock Holmes in The Sign of Four "How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?”
Along the way, you may need tools such as OpNet to help isolate every component of hardware and software and gather data. In this case, our usual network-based software tools did not help because the critical connection we needed to check was the traffic between the phones and the interface boards on the paging server. We needed to analyze if the DTMF input coming from our telephone system was providing the correct digits. If the digits captured before hitting the interface boards were bad, we would know it was how the DMTF signals were handled on our side. If the digits differed from what was being logged in the paging server, we would know the problem was inside server.
I wish troubleshooting intermittent IT problems was easier, but alas, many modern technologies have so much complexity that it takes all the skills of forensic pathologist to solve the problem.
The folks at CSI would be proud of my team.
Here's a brief example of the efforts we employ to solve IT mysteries.
A few weeks ago we were told that the Interpreters at BIDMC often received pages with ambiguous call back numbers. At BIDMC, valid numbers are 5 digit extensions, 7 digit local numbers and 10 digit long distance numbers. Interpreters often received 4 digit or 6 digit numbers that were impossible to call back.
The most obvious explanation for such an intermittent problem that only seemed to occur in one department was human error. Doctors misdialed numbers, assuming the last 4 digits would be enough to identify their call back number.
We sent out a broadcast email instructing the clinicians to always dial at least 5 digits.
That did not cure the problem.
We then began a data analysis. Could we relate the bad pages to a particular individual, department or location. We found no correlation.
We then asked if the problem was truly isolated to Interpreters. Our data analysis suggested that it occurred regularly in several departments. No others had mentioned it, but the problem was real.
We then asked if the problem was unique to BIDMC, since we share a paging system with other hospitals. The analysis suggested that it was unique to us, since other hospitals did not have the problem.
It seemed unlikely that just our doctors were using the paging system improperly, so we began analyzing all the hardware involved in paging - phones, interface boards, servers and software. Since some of these components were redundant we experimented with taking one member of clustered services offline to see if we could isolate a problem in one switch, one signal processor or one server. Still no resolution.
We then spoke with the manufacturer of the paging software. They had no reports of similar problems. When then spoke to engineer who wrote the software. He had an idea.
When you use a paging system, the dialog goes something like this
"Please dial the person you wish to page"
User enters pager number
"Please enter your call back number"
User enters call back number
"Thank you"
Page is sent
Many interactive telephone systems have a "buffer clearing function" that clears any input before users enter numbers to eliminate digits being carried over between voice prompts and to cancel out any background sounds that might have occurred between prompts. This sounds great, but what happens if an experienced user begins to dial immediately without listening for the prompts i.e. a person dialing very fast, just immediately enters the pager number, then pauses for 1 second before the "Please enter your callback number" prompt is completed, and enters the call back number. It's likely call back number digits would be truncated.
We turned off buffer clearing and here's what happened.
------Percent Bad Call Back Numbers------
Pager Pre-change Post-change
Russian interpreter 163/493 = 33% 3/244 = 1%
Spanish interpreter 262/959 = 27% 4/404 = 1%
Chinese interpreter 207/1086 = 19% 8/417 = 2%
Of the 15 pages in the post-change group listed as bad, most, if not all, were likely miss-dialed. For example, a couple were "123". Others were bad, but were immediately followed by what appeared to be a correct return page suggesting the caller knew they had entered bad data.
The troubleshooting process was complicated by the fact that truncation only occurred with pagers signed out to other numbers as is the case with Interpreter Services. The "clear buffer" had no impact on pagers not signed out. We tried many times to "type ahead" call back numbers for these and were unable to mimic the problem. It appears to have only been a problem if someone signed their pager out to another. It turns out that the buffer clearing software works differently for pagers with a status of "signed out/covered by"
My advice for diagnosing complex operational IT problems is to work stepwise with every layer of architecture, and isolate the problem. Then, follow the advice of Sherlock Holmes in The Sign of Four "How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?”
Along the way, you may need tools such as OpNet to help isolate every component of hardware and software and gather data. In this case, our usual network-based software tools did not help because the critical connection we needed to check was the traffic between the phones and the interface boards on the paging server. We needed to analyze if the DTMF input coming from our telephone system was providing the correct digits. If the digits captured before hitting the interface boards were bad, we would know it was how the DMTF signals were handled on our side. If the digits differed from what was being logged in the paging server, we would know the problem was inside server.
I wish troubleshooting intermittent IT problems was easier, but alas, many modern technologies have so much complexity that it takes all the skills of forensic pathologist to solve the problem.
The folks at CSI would be proud of my team.
Monday, December 22, 2008
Next Steps for Interoperability
There are some folks in Washington who have made statements that we should delay investments in EHRs because current vendor products lack the functionality needed to support a coordinated healthcare system. Others have said that we lack the standards or security framework to implement interoperability. Here are my thoughts.
Take a look at the successes in Massachusetts and New York with commercial EHR products. We've implemented eClinicalWorks, which includes decision support, e-prescribing, administrative transactions with payers, clinical summary sharing across the community, and quality measurement (all the National Quality Forum high priority measures). It's web-based, using a service oriented architecture in a cloud computing environment. By implementing this product at BIDMC, we're meeting all the payer guidelines for delivering appropriate, coordinated, high value care. Vendor products from Epic, Allscripts, NextGen, GE, Meditech, eMDs, MedSphere, and other CCHIT certified vendors have similar features.
Should we wait for something better that has more interoperability?
Do you drive a car? Why? It pollutes, costs a lot, and generally is not very efficient in traffic. You'd be much better off asking Scotty to beam you up via the transporter. Should we eliminate all cars, planes and trains until the transporter is invented? The same can be said of EHRs and health information exchange. My definition of good enough includes:
*Support for medication interoperability such as e-prescribing linking providers, payers, and pharmacies
*Support for laboratory and radiology interoperability such as orders and results integration among providers, hospitals and commercial labs
*Support for seamless electronic interchange between providers and payers for administrative data flows.
In 2009, several EHR vendors will support clinical summary exchange.
We can achieve a substantial improvement in care quality and coordination by implementing the systems available now and not waiting. If anyone thinks writing a next generation interoperable EHR from scratch is a good idea, have them look at the UK implementation of the NPFit/Connecting for Health project. They hired numerous companies to implement an new scheduling/booking system, a nationwide PACS system and a coordinated health record. After spending billions, they have limited success and low provider satisfaction.
On December 18, HITSP completed all the national standards harmonization work for 2008. This included:
*Biosurveillance Interoperability Specification (IS02)
*Consumer Empowerment and Access to Clinical Information via Networks Interoperability Specification (IS03)
*Emergency Responder Electronic Health Record Interoperability Specification (IS04)
*Consumer Empowerment and Access to Clinical Information via Media Interoperability Specification (IS05)
*Personalized Healthcare Interoperability Specification (IS08)
*Consultations and Transfers of Care Interoperability Specification (IS09)
*Immunizations and Response Management Interoperability Specification (IS10)
*Public Health Case Reporting Interoperability Specification (IS11)
*Patient-Provider Secure Messaging Interoperability Specification (IS12)
*Remote Monitoring Interoperability Specification (IS77)
The documents are accessible through www.hitsp.org
This latest round of work means that we've completed the three year AHIC roadmap for standards. There are no unapproved standards at this point!
Of course standards will evolve and we'll keep enhancing this work, including lessons learned from implementation in vendor products. In 2009, we'll be given a new body of work including Newborn screening, and filling several small gaps required to support clinical workflows.
Thus, if we have products that are good enough and interoperability standards, what are we lacking? Some say security.
HITSP completed security standards harmonization in 2007.
The 2008 CCHIT criteria for security are rigorous. Vendors have described them to me as one of the most challenging aspects of certification.
Although there is still local/state variation in policies, we do have a national framework for EHR and PHR data exchange.
Some say that they have personal experiences with lack of coordinated care among multiple providers. Is that an issue with EHRs and standards? My view is that this is a process and policy issue. In the US we do not have a healthcare system, we have numerous providers, labs, pharmacies, and hospitals which do not constituent a single medical home for the patient.
Let's implement EHRs now and realize their benefits. Let's implement the interoperability for administrative transactions, labs/rads, and e-Prescribing that is robust today. Then let's implement the clinical summary exchange that's coming soon. It's a journey and we should start immediately. There is no reason to wait.
Take a look at the successes in Massachusetts and New York with commercial EHR products. We've implemented eClinicalWorks, which includes decision support, e-prescribing, administrative transactions with payers, clinical summary sharing across the community, and quality measurement (all the National Quality Forum high priority measures). It's web-based, using a service oriented architecture in a cloud computing environment. By implementing this product at BIDMC, we're meeting all the payer guidelines for delivering appropriate, coordinated, high value care. Vendor products from Epic, Allscripts, NextGen, GE, Meditech, eMDs, MedSphere, and other CCHIT certified vendors have similar features.
Should we wait for something better that has more interoperability?
Do you drive a car? Why? It pollutes, costs a lot, and generally is not very efficient in traffic. You'd be much better off asking Scotty to beam you up via the transporter. Should we eliminate all cars, planes and trains until the transporter is invented? The same can be said of EHRs and health information exchange. My definition of good enough includes:
*Support for medication interoperability such as e-prescribing linking providers, payers, and pharmacies
*Support for laboratory and radiology interoperability such as orders and results integration among providers, hospitals and commercial labs
*Support for seamless electronic interchange between providers and payers for administrative data flows.
In 2009, several EHR vendors will support clinical summary exchange.
We can achieve a substantial improvement in care quality and coordination by implementing the systems available now and not waiting. If anyone thinks writing a next generation interoperable EHR from scratch is a good idea, have them look at the UK implementation of the NPFit/Connecting for Health project. They hired numerous companies to implement an new scheduling/booking system, a nationwide PACS system and a coordinated health record. After spending billions, they have limited success and low provider satisfaction.
On December 18, HITSP completed all the national standards harmonization work for 2008. This included:
*Biosurveillance Interoperability Specification (IS02)
*Consumer Empowerment and Access to Clinical Information via Networks Interoperability Specification (IS03)
*Emergency Responder Electronic Health Record Interoperability Specification (IS04)
*Consumer Empowerment and Access to Clinical Information via Media Interoperability Specification (IS05)
*Personalized Healthcare Interoperability Specification (IS08)
*Consultations and Transfers of Care Interoperability Specification (IS09)
*Immunizations and Response Management Interoperability Specification (IS10)
*Public Health Case Reporting Interoperability Specification (IS11)
*Patient-Provider Secure Messaging Interoperability Specification (IS12)
*Remote Monitoring Interoperability Specification (IS77)
The documents are accessible through www.hitsp.org
This latest round of work means that we've completed the three year AHIC roadmap for standards. There are no unapproved standards at this point!
Of course standards will evolve and we'll keep enhancing this work, including lessons learned from implementation in vendor products. In 2009, we'll be given a new body of work including Newborn screening, and filling several small gaps required to support clinical workflows.
Thus, if we have products that are good enough and interoperability standards, what are we lacking? Some say security.
HITSP completed security standards harmonization in 2007.
The 2008 CCHIT criteria for security are rigorous. Vendors have described them to me as one of the most challenging aspects of certification.
Although there is still local/state variation in policies, we do have a national framework for EHR and PHR data exchange.
Some say that they have personal experiences with lack of coordinated care among multiple providers. Is that an issue with EHRs and standards? My view is that this is a process and policy issue. In the US we do not have a healthcare system, we have numerous providers, labs, pharmacies, and hospitals which do not constituent a single medical home for the patient.
Let's implement EHRs now and realize their benefits. Let's implement the interoperability for administrative transactions, labs/rads, and e-Prescribing that is robust today. Then let's implement the clinical summary exchange that's coming soon. It's a journey and we should start immediately. There is no reason to wait.
Subscribe to:
Posts (Atom)
