Cool Technology of the Week

As readers of my blog know, I drive a Toyota Prius Hybrid which includes built in GPS navigation and Bluetooth links to my Blackberry. I drive about 30,000 miles a year to customer sites and to various speeches throughout New England. The challenge in the Boston metro area (and in most metro areas) is that a GPS navigation system is only one small part of the puzzle. Traffic congestion can be so bad that a 5 mile commute can take over an hour. On December 21, traffic in the Boston area was so bad that many folks took 5-7 hours to go 10 miles from downtown to the suburbs. That night I commuted from Washington DC to Wellesley in 6 hours simply because the airport is directly connected to the Mass TurnPike via the Big Dig, avoiding all congested side roads.

Every morning I need to decide which route to take to the office, guessing about traffic congestion points based on radio news reports that are often wrong or not current.

My cool technology of the week is Google Maps with Traffic for Blackberry

This morning, I downloaded the application from Google to my Blackberry 8707G while driving and instantly got a map of all traffic congestion points in the Boston area. I made it into the office in record time. I took a photo (above) of my Blackberry running the application. You'll see traffic patterns in green, yellow and red. Congestion points are Route 93 and the intersection of Route 128/Mass Pike.

The application is a relatively thin client - it took about 1 minute to download and includes live traffic updates for 30 major US metropolitan areas, movable maps with satellite imagery, directions, and business locations. Interactive maps let you zoom and move in all directions so you can orient yourself visually.

I look forward to the day when the Prius includes this functionality, but for now, Google Maps with Traffic for Blackberry will save me up to an hour a day, making me more productive.

Trapped in the 9th Circle of Verizon

For those who are not familiar with the reference to Dante's Inferno, let me summarize without being profane. The opposite of "Heaven" is the Verizon Customer Service infrastructure.

On Thursday, December 20, my FiOS internet/TV service was shut off by Verizon without any notice or warning. Verizon has the best wireless and wired technology in New England, but not the best customer service. Since I receive 4 separate paper bills every month (we've tried to get them electronic for a year, more on that later) for my home Verizon phone, internet, TV, and wireless services, I was fairly confident that Verizon could not figure out where to credit my payments, so my wife called their customer service number. What follows is a tale involving an army of customer service representatives, 4 hours on the phone, and 24 hours to restore my service.

December 20, 2007
3:00pm My wife turned on the television and it displayed "no channels available". Verizon did not call, send us a dunning letter or attempt to contact us in any way before terminating our service. My wife went to the Verizon website and found the customer service number 1-888-553-1555.

4:00pm After she held for some time, the friendly gentleman that finally came on the line said that because we lived in Massachusetts, he (inexplicably) could not actually help. He transferred her to another number 1-888-338-9333 .

4:30pm Another courteous gentleman came online, and because my wife did not have our FiOS account number available, he eventually indicated he could not locate our account number with only name, address and phone number, so he would have to transfer her to someone within the FiOS Repair Order area that had access to a different kind of customer relationship management system.

4:45pm Another kind gentleman came on line from the repair area and indicated that he could find our account, and that FiOS service had been switched off due to the mailed funds not being applied, but he could not help her with the problem since he was a repair area service facilitator. He gave my wife the number of Verizon Financial Services 1-888-250-4909.

5:30pm She waited on hold for a representative for 35 minutes. When Tiffany answered, she was clearly not pleased with my wife's lack of knowledge of our FiOS account number. She told her that the only way to pay the bill was to dial the automated payment system (no humans) at 1-800-345-6563 using the FiOS account number. She gave my wife a 9 digit alphanumeric account number but unfortunately the automated payment system needs an 18 digit account number. The automated payment system recommended another number 1-866-326-7937.

6:00pm A pleasant woman came on the phone and found our FiOS account number with no difficulty using our phone #, and reiterated that we must call the automatic payment number at 1-800-345-6563, be sure to get the receipt number, and then call 886-438-3467 to reinstate service, but they had of course already closed shortly before at 6pm Eastern Time.

6:30pm My wife called 1-800-345-6563 for automated payment and obtained a receipt number for the payment, but then had to wait until 8 am the following day to call for reinstating.

December 21, 2007
8:00amMy wife called 1-886-438-3467 and through good karma, the phone was answered by Renee, who trains others for Verizon and has a broader working knowledge of Verizon workflow. She found our account, noted that all past due balances were paid and that we did not need the receipt from the automated payment system. She also noted that our Verizon accounts have non-matching social security numbers in the phone number and the FiOS accounts. Somehow, Verizon does not believe that Dr. and Mrs. Halamka at the same address are related.

Renee initiated a three way conference call with me on my Verizon cell phone and my wife on our home phone to verify our identities and social security numbers. She then began the long and complex, multi-division process to create one bill. She also got the FiOS division on the phone and confirmed our service was reinstated.

Renee planned to call back after the holidays, and after all the various work orders went through their system, planned to create an integrated billing package for us. Unfortunately, she told us that she could not give out her full name, email address, or phone number to call her back.

January 2, 2008
5:00pm Renee deserves an immense amount of praise - she called me back today, the first Verizon employee to ever followup with us. She noted that she put in the order to consolidate all our accounts but noted that one of our cell phone accounts had the zip code 02482 instead of 02481. She agreed to place a conference call to Verizon Wireless because she is not empowered to change zip codes. She even tried to save us money by putting us into one of the national service rollup plans. All sounded perfect, until she checked her computer and noted that none of the consolidation orders she entered seemed to have processed. I have complete faith in Renee, and her tenacity may end up resolving these issues over the next few weeks.

Meanwhile, today I received the following mysterious billing notice from Verizon:

"Regarding your Telephone, we recently received a request to change the billing address on your account".

Who knows what that means? Maybe they'll move my phone service next door?

There's another great Catch 22 with Verizon. We've tried for a year to enable electronic payments so that bills are automatically paid without having to worry about paper, the US Mail or figuring out what account to credit. Each month, we fill out the paperwork but still receive a paper bill the next month. According to the electronic billing policy, if you pay the paper bill, your election to pay electronically is instantly voided. If you don't pay, your service is shut off. Rock or Hard Place, Frying Pan or Fire? Your choice.

All I ask of Verizon is a single, obvious phone number to call, hiding the complexity of the company from the customer. Maybe, they could call this idea One-Bill (R). When I call that service number, a human should be empowered to take action. As CIO of Harvard Medical School and CareGroup, I spend millions every year with Verizon and I cannot navigate Verizon Customer Service. If anyone at Verizon reads this and cares about customer service, please feel free to make this blog entry a case study. I'm sure dozens of broken processes could be identified just by highlighting my experience.

I'm off to write 4 checks for my 4 paper Verizon bills now, but I feel better and at least the TV works.

Disaster Recovery Planning

In response to my posting about IT Governance, I received a very good question about prioritizing infrastructure spending: "Without an IT infrastructure steering committee, how do you resolve investment prioritization around these unseen but critical investments?"

Every year, I receive approximately $10 million dollars at BIDMC and $3 million at HMS for infrastructure spending on networks, servers, desktops, storage and wiring. This budget is an annuity based on the value of our IT infrastructure and the lifecycle of the components. However, it does not include funding for disaster recovery.

Five years ago, an audit at BIDMC pointed out our vulnerability to a disaster affecting the CareGroup data center, since the building itself is a single point of failure. I worked with the Board and senior management to raise awareness of disaster recovery planning and the need to make a multi-year capital investment. I've mentioned our disaster recovery work in previous blog entries, but not provided the details.

Cost of Information Technology
What will keep me up at night in 2008
Some Like it Hot

Here are all the details of how we're doing it including our budgets.

BIDMC
Step 1 We inventoried all our applications and determined the service levels required based on the business impact of downtime. We did not hire a team of expensive consultants for a formal business impact analysis. Instead, we used our existing governance committees to brainstorm how long applications could be interrupted before clinical workflow would be disrupted to the point of causing harm. Here are some examples of our informal business impact analysis:

Code Paging system - If a patient suffered a cardiac arrest and the code team did not respond, the patient could die. Hence, downtime of the code paging system must be a few minutes per year at most. No downtime at all is the goal.

Provider Order Entry - If medications, diagnostic testing and diets cannot be ordered, patients could have delays in therapy resulting in pain, extended illness or harm. Hence downtime of POE must be hours per year at most.

Revenue Cycle systems - If bills cannot be sent out for a day, no real harm is done since billing in hospitals is not a real time activity. However, if several days pass without billing, cash flow could be interrupted. Hence, downtime of revenue cycle systems must be a few days per year at most.

Library catalog - If the library catalog is disrupted, users will have to seek other sources of information on the web. A slight inconvenience will occur. Downtimes could be extensive without causing harm.

Step 2 We mapped out single points of failure in power, cooling, networks, servers, storage and infrastructure applications (i.e. DNS/DHCP) We developed an incremental plan to address these vulnerabilities and hired a new employee to coordinate risk mitigation efforts, beginning with enhancements to our existing data center.

Step 3 Since the data center itself was a single point of failure, we constructed a geographically distant data center to mitigate loss of the primary data center and have begun replicating data and applications in this secondary location.

BIDMC is in year 3 of a 5 year disaster recovery center implementation plan. The year by year budget totaling $13 million dollars which will support the recovery time and point objectives specified by our business impact analysis is here.

Harvard Medical School
HMS is in year 2 of a 5 year plan to provide similar protections. Since HMS is not a healthcare delivery organization but provides education, research and administrative services, the uptime requirements are less rigorous. HMS had a slightly different set of business requirements to meet when we began this project. Its primary data center was located in a 100 year old building with limited electrical and cooling support. Hence we wanted to establish a secondary data center which was an extension of the existing primary data center, then move all mission critical systems to the new data center, reserving the original data center for less critical applications and disaster recovery. The HMS five year milestones can be summarized as

Year 1 Create a new Data Center and run both the old and new physical locations as a single "virtual data center". This allowed us to keep existing applications running, add new applications to the new data center, and migrate servers from old to new in phases.
Year 2 Create a redundant network core and begin to operate the two physical locations as a primary and backup data center. Hire a disaster recovery coordinator.
Year 3 Create redundant storage, high performance computing and active/passive email hosting divided between the two data centers.
Year 4 Create redundant installations of critical applications between the two data centers
Year 5 Create redundant installations of critical applications between the two data centers

Each year of these plans enables us to progressively reduce risk. Of course, this disaster recovery planning must be complemented by a disaster response plan including calling/paging trees, communication strategies, and a playbook for responding to critical incidents. I'll post these plans in a later blog entry.

Just like security, disaster recovery planning is a journey. It requires a dedicated team, a project plan and a budget. We'll never be done, but by 2011 we'll have mitigated the risk of single points of data center failure for the majority of our applications.

My New Year's resolutions

It's the time of year to synthesize the lessons learned from the past year and think about new approaches for the coming year. Remember - the definition of "insanity" is doing the same thing in the same way and expecting a different result. Here are my 2008 New Year's resolutions:

BIDMC
We're implementing a $4 million hosted electronic health record solution for our non-owned doctors, subsidizing 85% of the start-up costs for clinician offices. The effort involves collaborating with several vendor and implementation partners (I'll provide all the details in an upcoming post). Only by subsidizing costs, providing the resources to transform practices from paper to electronic workflow, and supporting clinicians post go live can we reduce the barriers to adopting electronic health records. My resolution is to reduce the total cost of implementing an EHR for private doctor's offices from the historical average of $40,000-60,000 to $25,000 per physician.

Over the past 10 years, I've been able to centralize all infrastructure purchases and support, phasing out most departmental IT spending. Currently we have just a few applications that are managed by departments. Over the next year I want to align the change management and communications processes for all applications, including those maintained by departments, with IT standard processes. My resolution is to create an ecosystem of change management processes followed by all.

Although we have strong governance processes, there are occasional projects that engage IT at the 11th hour, imposing unrealistic deadlines for phone, network, and desktop support. My resolution is to develop and widely communicate guidelines to all stakeholders imposing a 90 day notification period on opening new locations needing network, telecom, and desktop support.

HMS
The Dean's strategic planning process is likely to create a demand for more collaboration tools. We'll migrate from Web 1.0 tools that focused on content management to Web 2.0 tools that focus on collaboration among users from many Harvard affiliated organizations. Another aspect of this is providing the tools to locate people, equipment and knowledge - call it a matchmaking service for 18,000 faculty. Maybe Harvard's version of eHarmony.com for science? My resolution is evolve our web frameworks to meet the needs of next generation collaboration.

My IT budget administrator is transitioning to another HMS position on January 4 and I would like to recast the position as the Business Manager of IT, producing all the data needed to make most IT purchases an annuity i.e. a desktop with a 4 year life results in operating budgets to replace 1/4 of all desktops each year. Even new FTEs can be algorithmically linked to growth in demand for support, storage, and high performance computing. My resolution is to create a highly predictable, data driven annual budget process.

The current scope of IT services at Harvard is infrastructure support (desktop, server, network, storage) and enterprise applications. However, we do not have the staff to support ad hoc web design, assist researchers with database creation or coordinate niche application programming in support of grants. My resolution is to extend the scope of IT services to include the application support that will empower researchers to get new grants which include innovative IT methods.

MA-SHARE
The Massachusetts Regional Health Information Organization (RHIO) provides Health Information Exchange services for e-Prescribing and secure clinical data sharing in Massachusetts. My resolution is to expand the number of hospitals, clinicians, and payers connected to this infrastructure.

NEHEN
The New England Health EDI Network, a regional financial data exchange, connects payers and providers with 100 million transactions per year. Over the the past year we've worked to make this advanced IT system available to every doctor in the state, even small solo practitioner offices. My resolution is to increase the number of small community practices using this technology.

HITSP
The Healthcare Information Technology Standards Panel has harmonized electronic data standards for 3 use cases in 2006, 5 use cases in 2007 and has been assigned 6 use cases for 2008. My resolution is to harmonize standards for consumer healthcare devices, remote consultation, referral management, the genome/family history, public health case reporting and immunizations by October 2008.

Since a blog can be a personal glimpse into the life of the author, here are my personal resolutions:

• Donate 10% of my income to fund technology for those on the other side of the digital divide
• Continue to reduce my carbon footprint by
  1. Eating regionally grown vegan foods, freshly prepared each day, eliminating the need for wasteful packaging
  2. Reducing my airline travel by 20%
  3. Consuming less by reusing, recycling, and renewing - the end result being less churn of my belongings
• Take walks with my wife and daughter (the picture above) through a local forest three times per week
• Play a Japanese flute concert on the top of Eichorn Pinnacle in Yosemite

I look forward to a great 2008 with all my staff, my customers, and the industry. A toast to all of you!

IT Governance

One of the most important steps a CIO can take to ensure alignment of IT with the business strategy of the organization is to create robust governance committees. It's also the best way for a CIO to satisfy customers, respond to the tyranny of the urgent, and keep the CIO employed!

I've mentioned governance issues in several previous posts:
Time Scope and Resources
How to Say No
Tyranny of the Urgent
It's not a Job it's a Lifestyle

In the interest of transparency, I'd like to describe my governance successes and failures plus my 2008 plans for IT governance.

At Beth Israel Deaconesss, I have committees for each of my major groups of IT customers:

• Laboratory Information Systems co-chaired by the Senior Vice President (SVP) for Operations and the Chief of Pathology (an MD)
• Radiology Information Systems co-chaired by the SVP for Operations and the Chief of Radiology (an MD)
• Critical Care Information Systems chaired by the Director Trauma, Anesthesia and Critical Care (an MD)
• Inpatient Information Systems (includes Provider Order Entry) chaired by the Senior Director of Clinical Resource Management (an MD)
• Ambulatory Information Systems chaired by the SVP of Ambulatory & Emergency Services (an RN)
• Health Information Management Information Systems chaired by the Director of the Hospital Medicine Program (an MD)
• Community Information Systems chaired by the Executive Director of the Physician's Organization and the SVP for Network Development (an MD)
• Decision Support Steering Committee chaired by the Director of Business Planning & Decision Support and the SVP forHealthcare Quality (an MD)
• Enterprise Resource Planning (ERP) Information Systems chaired by the Director of Business Services and the Controller
• Revenue Cycle Information Systems chaired by the Chief Financial Officer

This structure worked very well for the past 10 years, ensuring that each application had a lifecycle prioritized by the clinicians and not the IT department. However, in 2007, we needed to make a change. As BIDMC grew into a 1.2 billion dollar organization, an emphasis was placed on achieving an operating margin which would yield the capital budgets needed for expansion. This meant that IT budgets did not grow at the same pace as the clinical budgets and led to competition for IT resources among my governance committees. Existing governance committees set the right priorities within each business area, but we did not have a governance construct to set priorities among all the business areas. Thus, we created an overall IT Steering Committee comprised of the chairs of each of the existing governance committees. The terms of reference for this new committee are here.

At Harvard Medical School (HMS), I also have committees for each of my major groups of IT customers:

• Administrative Information Technology chaired by the Executive Dean for Administration
• Educational Applications Committee chaired by the Executive Director of Curriculum Programs
• Research Information Technology chaired by the Director of the Research Information Technology Group

Like BIDMC, these three committees functioned very well over the past 5 years to ensure priorities were set within the domains of the three core businesses of HMS - research, teaching and administration. A new Dean of HMS took office on September 4, 2007 and launched a strategic planning process. The result of this process could be a substantially broader scope for IT, requiring new resources and scalability. Depending on the outcome of the planning processes, IT governance may need to be revised. Harvard University just completed a governance audit of IT departments and the following are the unedited conclusions about Harvard Medical School:

"A school-wide committee overseeing coordination of IT resources among HMS’ three primary business groups does not exist. HMS has functioned as three core businesses: research, education and administration. HMS IT has established governance processes for each of these three businesses which have led to a high degree of customer satisfaction.

As the new HMS strategic planning process creates new projects and stakeholders, the individual governance committees will evolve to align with the new strategic needs, including the creation of a school-wide IT Steering Committee if appropriate. There is a risk that IT resources could be allocated inequitably among the three core businesses and decisions made without the involvement of key business stakeholders.

The HMS CIO will participate in HMS strategic planning, identifying and documenting governance requirements and school-wide committee needs to ensure appropriate allocation and prioritization of IT resources by May 1, 2008. "

Thus, there may be a need for an overall IT Steering Committee at HMS. Bigger committees are not always better committees and creating a committee to objectively balance the heterogeneous needs of research, education and administration will be challenging. However, I'm very willing to do it if the demand for resources by any one group of customers significantly conflicts with the requirements of other customers.

A few lessons learned from the governance experience above:

In a hospital, it is key that clinicians (MDs and RNs) run the IT governance committees. You'll note that I do not chair any committee other than serving as co-chair of the overall steering committee. My role in that committee is a facilitator only and I do not vote on priority setting.

It's very important to have governance committees that are focused enough to really grasp the details of stakeholders needs. It may appear that I have too many governance committees, but this is the parsimonious number required to ensure that priorities are set at the application level.

Governance must evolve with the needs of the business. I am a servant of the organizations which employ me and I do not have an agenda of my own. Hence I will gladly change governance as needed to be maximally responsive to changes in the business environment around me.

I want to thank the Harvard Risk Management and Audit Services for their work this Fall which truly enabled me to evaluate the effectiveness of all my IT governance groups.

Cool Technology of the Week

Over the past 2 months, I've been evaluating technologies to support flexible work arrangements such as working from home. I've tried MSN messaging, Yahoo IM, AOL AIM, Second Life, Wikis, Blogs, Facebook and Webex. Each one of these sites required me to establish a new user account. To be honest, I cannot remember which username and password is used with which site. OpenID is the cool technology of the week that can help solve this mess by creating "single sign on" across many vendor products.

The idea is simple - a web site serves as a trusted site for OpenID credentials. Other websites then trust this site, using it to authenticate users via simple well known internet standards ((URI, HTTP, SSL, Diffie-Hellman). By using OpenID, websites such as AOL, Technorati, Blogger, and Plaxo make it easy to signup and login, empowering users with one credential for all their instant messaging, blogging and social networking needs. The complete directory of all internet applications which support open ID is here. It's estimated that there are over 160-million OpenID users with nearly ten-thousand sites supporting OpenID logins.

There are caveats. Anyone can sign up to be a source of OpenIDs, so an unsuspecting user may sign up for credentials on an inscrutable site. Once their OpenID credentials are known, they could be used to by a hacker to break into banking or other sites not specifically OpenID enabled, since most users tend to reuse similar credentials at every site they access. There is no concept of certifying an OpenID provider or running a criminal record information check on folks who operate OpenID sites.

That being said, the OpenID, is certainly useful for those sites where security and identity pose little risk such as social networking and informational web sites. Also, OpenID could be very useful for intranets, where the provider of the OpenID is the institution itself and users then use OpenID to access applications running within the institution. In my next revision of the Harvard portal called eCommons, I will support OpenID as a means of linking together all the various domain credentials used in the Harvard environment.

In my opinion, the internet will eventually move to the concept of federated trust for authentication such as OpenID. OpenID will become even more powerful and useful when there is a credentialing mechanism to certify providers are trustworthy.

It's all about workflow

On occasion, the business owners I serve suggest that new software will solve all their workflow problems. Time and time again, we learn that it's not the software that really matters, but good processes. Automating a broken workflow does not achieve a positive result. Re-engineering workflow, then automating it, results in a successful project for everyone.

Since it's the day after Christmas and many people are rushing to malls for after Christmas sales and returns, here's a seasonal tale of my recent experience with workflow from an IT perspective.

My wife asked me to return a few holiday items to Target. They had an efficient queuing system set up to enable four clerks to serve a well ordered line. The process is simple - hand the receipt to the clerk, then hand the items to the clerk. Each receipt is archived for 90 days and has a unique bar code at the top. The clerks do not need to read the receipt, they simply scan the bar code and all the items are retrieved into a local cache. The clerk then scans each returned item and it is checked against the local cache for price, verification of purchase, and the fact that it has not already been returned previously. This prevents fraudulent return of items not purchased from Target. Most importantly, Target has decided that this verification workflow is all that is needed to return an item. No manager/supervisor approval is needed, no key is used to open a register and no credit card is needed. All returns are automatically credited against whatever method of payment was used for the original purchase. By empowering the clerks to process returns this way, the customers are very satisfied, no manual keying of data is needed so accuracy is high, and I'm motivated to buy again from Target, knowing that I can easily return anything I purchase.

My wife also asked me to return 10 extra towels/linens to Bed, Bath, and Beyond. As I entered the store it was clear that the workflow was broken. I found no orderly queue and unclear responsibilities as to who provides specific customer services. I found a very helpful enthusiastic employee who began to manually match the 16 digit bar codes on my receipt with the bar codes on each towel to verify that I had the correct receipt. Once she manually circled each bar code and initialed them, she then scanned them into the register. She was not empowered to actually process any return transaction, so after 20 minutes of manual paperwork she then paged a manager. The manager was busy so he suggested the supervisor, who was busy ringing up new sales. After trying to interrupt the supervision unsuccessfully, it was clear that another page to the manager was necessary. This time, the manager responded, reviewed the bar codes on each of the towels again, checking the receipt again, then inserting a key in the register to enable a return. I then was asked to produce the original credit card used so that it could be credited. Luckily I had a copy of my wife's Visa card with me. Finally, after 30 minutes, 3 people and manual paperwork, my 10 towels were credited and the $50 dollars was placed back on my credit card. I'm reluctant to purchase from Bed, Bath, and Beyond again, since I know any return will take more of my time than I have available. Considering the time and gas involved, it would have actually been more cost effective to donate the towels to a worthy cause. The very nice folks at Bed, Bath, and Beyond said that IT was working on a software solution for 2008. Let's hope they re-engineer the workflow first to empower clerks to process returns!

So next time you're told that software will solve the customer's business process problems, be sure to study the workflow first!