At Home with Hurricane Irene

For the past 2 years, I've operated a weather station which provides realtime data for Wellesley, Massachusetts to the National Weather Service, the Citizens Weather Observation Program, and Weather Underground.

If you search Google for Weather Wellesley, you'll get my data.

For graphs of the temperature, barometric pressure, wind speed, wind direction, and rainfall rate during Irene, here's a summary from Weather Underground.

To prepare for the storm, I took down flags, removed hanging bird houses, and stored every object that could become a projectile in the wind.

Interestingly, we never had sustained winds more than 10 mph.  Our peak gust was 17 mph.    Likely, the impact of the storm on my location was much less than the surrounding neighborhood because of the grove of old hemlocks nearby that serves as a windbreak.

However, we did have a substantial amount of rain - 2.24 inches on Sunday and 4.66 inches in 24 hours related to Irene.

I retrofitted all our gutters with extenders to push water away from the house and sandbagged the bulkhead to our basement, just in case standing water accumulated in the backyard.

The key technology that saved our house was not anything wind related, but was the disaster recovery sump pump I created last year.

At the height of rain intensity, a tree down the street collapsed. due to the weight of water on its leaf canopy, and fell through power lines.   The Department of Water and Power cut power to the neighborhood to do the repair.

Water began rising in my basement drains and as designed, the battery backup sump pump worked perfectly, pumping the basement dry despite the loss of power.

Installing a battery backup sump pump makes great sense - the likelihood is that in the worst storms, you'll also lose power, so having an AC powered sump pump will not help you (unless you engineer a complete alternative power solution for your home)

If my neighborhood was typical, the storm did have a profound impact.   Down the street, a tree fell into two cars.   My father in law lost a portion of his entry roof.   Downed limbs have impacted traffic flows throughout the area.

However, our engineered systems for power backup and water control plus our preparation for the storm made our experience of Irene, our first hurricane, uneventful.

Cool Technology of the Week

I've written about strong identity management and the use of biometrics for secure applications such as prescribing controlled substances.

Bio-key, a web-based biometric provider we have used at BIDMC, has now developed an iPhone and iPad finger print reader that does identity verification in the cloud.

Although I've argued with the FDA that SMS messages sent to clinician cell phones should be enough for 2 factor authentication, their response has been that doctors cannot use Blackberry's, they must use Barackberry's - fully encrypted highly secure devices when writing e-prescriptions.

Bio-key's approach to two factor authentication on iPhones and iPads will enable a new level of functionality and productivity for clinicians who want to use these consumer platforms for healthcare applications.   Today, over 1000 physicians at BIDMC use iPads and they are becoming the mobile device of choice for clinicians.

A cloud-based biometric authentication system for iPhones and iPads.  That's cool!

Preparing for the College Transition

In one week, we drop off our daughter to Tufts University so she can began the next era of her life as a college woman.

All of us have been preparing.

High School is a time of many emotions - high highs and low lows.   It's about discovering independence, making choices, accepting responsibility, developing relationships, and balancing parental authority with the desire for autonomy.

More is expected of today's teens than in my generation.  It's very stressful on a young person.

In one week, she'll make decisions on her own.  She'll decide what to eat (and drink), when to study, and who to spend her time with.

Over the past few weeks, she's thought about her transition in a very spiritual way.

I did not approach my college transition formally.   I packed my clothes and typewriter the night before and we drove from Los Angeles to Stanford for the drop off.     That was 31 years ago this week.

She realizes that she has to prepare for this new era while bringing closure to her childhood growing up in Wellesley, Massachusetts.

She has thought about all her Wellesley relationships.   She's scheduled events with every one of her friends to create positive memories and energy before they go their separate ways.    She's arranged hikes, picnics, movies, meals, and sleepovers.

She's taken private walks to her favorite places in Wellesley.    She's also made a conscious decision not to visit many of the places she treasured when very young so that she can remember them as they were from a child's point of view.

Yes, she'll stay in touch with friends on Facebook, but that will fade as she develops new relationships, new interests, and new goals.   The closure she's bringing now will leave lasting memories among all her friends, creating a sense of optimism and energy for the future ahead.

My wife and I know that next Wednesday will be hard.   We'll bring our daughter's carefully packed belongings (4 small bins that will fit perfectly in a cozy dorm room) to her new living space, set up her IT infrastructure (the home CIO at your service), and attend a formal matriculation ceremony.   My wife and I will give her the space she needs to bond with her new colleagues and we'll retreat to a quiet vegan cafe to reflect on the next era in our lives.

We've already planned a few short trips together.  My wife will join me for keynote addresses in Burlington Vermont, Phoenix Arizona, and London England.    We've already planned a family get together on Mt. Monadnock over Columbus Day weekend.    We've thought about the next few months and years as we've considered the implications of staying close to our daughter, our parents, and our jobs.

The end result is a solid plan that will launch all of us into the next stage of life.   For my daughter, it's adulthood.   For my wife and I, it's a refocus on each other, the world around us, and our careers.   The past 18 years with our daughter have been a gift, but the next era will be positive for all of us too.   Our evolution begins next Wednesday.

Storage Dreams

As I continue to support the infrastructure requirements of the research faculty of Harvard Medical School (in parallel with the process to find my own successor at HMS),  I have a storage dream.

The scene opens to a researcher logging into "Storage Central", a browser neutral, operating system neutral website that even runs perfectly on an iPad.

After thoughtful analysis of faculty needs, Harvard Medical School will have concluded that there are 3 different directory types in 3 different storage workflows

Directory types
a.  Massive numbers of small files (i.e. next generation sequencing) that needs solid state metadata management (i.e Isilon 32000X SSD)
b.  Small numbers of really large files (i.e. image processing) that needs high I/O throughput (i.e. Isilon 72000X)
c.  Average numbers of average sized files that can use lower performance technologies (i.e. Isilon 72NL)

Workflows
a.  Files with a high turnover rate (scratch space) that are created and destroyed daily.  No snapshot or archival tier is needed
b.  Files with a low turnover rate that do not need replication because the data is easy to regenerate. Snapshots are needed to protect the data against drive failure.
c.  Files with a low turnover rate that need to be retained for years due to compliance requirements and the difficulty of regenerating the data.  An archival tier is needed. (i.e. arrays of inexpensive 2 Terabyte drives)

The researcher sees a visual representation of her storage use in each directory and workflow, both currently and monthly over the past year.   Data on primary storage, snapshots used to protect the data, and archival copies of the data are shown separately.

The researcher oversees several post docs.   By clicking on a link, the researcher can see the storage use of all those she supervises.

Each directory type has a fixed three year cost per terabyte.   Workflows with snapshots or archives have an incremental cost.   These costs are well known and accepted by all the users.

The researcher can set their own quotas for directory types and workflows.   A calculation of cost for current storage and total quota is shown.   The researcher can type in a grant number or departmental account number to reserve the directory types and workflows they need.

The departmental administrator oversees many researchers.    She can view the storage use of all her faculty with historical, current, and projected costs shown on screen.

She can discover who is likely to exceed their budget and who is responsible for the largest amount of storage growth over time.

An IT storage concierge is assigned to each department to help researchers and administrators move data among directory types and workflows to balance performance and cost.    There is complete transparency between the demand created by the users and the supply provided by the IT department.  

The Dean knows the total costs charged to departments, the IT department, and the school (as overhead components in indirect costs).

The CIO and the infrastructure team receive daily summary reports which forecast growth so that additional storage can be added as necessary, ensuring that each directory type and workflow always has 20% unused capacity.   Storage vendors can ship nodes to expand each directory type and workflow within 1 week of receiving a PO, so storage can be expanded just in time without risking over or under provisioning.

The chargeback model is NIH compliant and motivates researchers to maintain files via the easy to use move/deletion tools in the web interface.

The research community, school administration, and  IT are deliriously happy.  Storage challenges are a solved problem.    The governance committees have turned their attention to cool applications that advance science instead of infrastructure limitations that impede it.

We're assembling industry experts to work on this dream.   My hope that is that I can report back in 2012 that the dream is now the Harvard Medical School reality.

Experience with Lion Part II

I recently wrote about my first experiences with Mac OSX Lion.

Now that I've been running Lion exclusively for a few weeks, I've learned a lot about my Macbook Air and the lifecycle of Apple products.

There are 4 variations of the Macbook air in use today:

Generation 1 - the 2008 Air with a sluggish 1.8" hard drive or an equally slow but expensive Toshiba SSD drive with 50 MB/s reads and 14MB/s writes.  It had a real world battery life of 2.5 hours.

Generation 1.5 - the 2009 Air that replaced the Intel GMA X3100 integrated Graphics Processing Unit with a Nvidia GeForce 9400M to support a 1280x800 pixel display.   The Toshiba SSD drive was replaced with a slightly faster Samsung 128 SSD.

Generation 2 - the 2010 Air that was SSD-only (Samsung 128C).  SSD performance improved beyond that of magnetic spinning hard disk drives. A new Nvidia GeForce 320M GPU enhanced graphics performance and the Air's screen resolution was increased to 1440x900 pixels.  The CPU was slower than in the previous models, but in practice it often performed better, because, unlike the old Airs, the newer ones didn't have to throttle down the CPU speed to keep the system from overheating.  Generation 2 included two USB ports, but peripherals were still limited by the maximum performance of the 480Mbps USB connections. In addition, Apple introduced an 11.6" model.

Generation 3 - the 2011 Air is based on the latest Intel Sandy Bridge Core i5 and i7 CPUs, which include hardware support for AES encryption and a Graphics Processing Unit on the CPU silicon.  The Mini DisplayPort connector which supported external displays in previous generations  was transformed into a Thunderbolt port, which drives external displays and provides I/O at 10 Gig/s.

I purchased my Macbook Air at the end of 2009, so I have a Generation 1.5 - a 2.13 Ghz Core 2 duo with 2G of RAM and a Samsung 128 SSD.

I installed Lion and fully encrypted the filesystem with Filevault2.

In practice Generation 1.5 does not have the CPU power and I/O necessary to sustain Filevault2 and application performance for I/O intensive operations such as Mail 5.0.

Here's a study of the I/O degradation caused by Filevault2 on the Generation 2 Air - a 44% decrease.   Generation 2 lacks the hardware AES encryption support (used by Filevault2) of Generation 3.

Generation 1.5 is even worse.  

The end result is that Mail 5.0  on my Macbook Air could not process the typical 1500+ emails I receive each day and encrypt/decrypt the filesystem simultaneously.   Deleted emails reappeared.   Emails that I moved between folders unmoved.    Only a reboot brought my Inbox up to date.

The solution - I reinstalled Lion without encryption and now Mail 5.0 works well, but running I/O intensive applications simultaneously like Skype 5.3 and Mail 5.0  is still problematic.

I do not store protected health information (or even personally identified information) on my laptop, so encryption is optional.

The Generation 3 Macbook Air with its I5 or I7, hardware AES support, and faster SSD drive is absolutely good enough for Lion, encryption, and I/O intensive applications.    However, the Generation 1.5 is not.    Running Lion and one application at a time is about all it can support.

Moore's law is alive and well at Apple,  with doubling of CPU capabilities every 18 months.   You should upgrade to Lion warily if you are running anything but the latest Air.

Healthcare is Different Part II

I recently posted a blog entry,  Healthcare is Different, examining the ways that healthcare differs from  other businesses.

Numerous folks sent me email agreeing and disagreeing with my points.

Here's a compilation of some additional ways that my readers suggested healthcare is different.

*Domain Expertise - the vocabulary, science, and physical skills necessary to practice medicine are very complex compared to most other professions.   For example to become a neurosurgeon requires kindergarten-high school, 4 years of college, 4 years of medical school, 7 years of residency, and generally a 2 year fellowship.   That's 30 years of education.

*No second chance -  In retail, if a good is defective it can be exchanged.  In service businesses, there is the concept of a redo, a repair, or renovation.  The concept of "returned goods" does not existing healthcare.

*Trainees.   There's probably no industry that is so inundated with "trainees" as health care; especially in an academic medical center.  They add a level of inefficiency during the learning process that is required to produce the next generation of health care workers.   In other industries, trainees come in small streams as you bring in co-ops, interns etc.    They don't come by the hundreds in July of each year.

*Highly regulated and compartmentalized workforce.   Healthcare has dozens of professionals whose practice is limited to certain privileges.   This inhibits mobility and cross-coverage that could improve the efficiency of the workforce.   If demand gets light in Cardiology, you can't easily move the clinicians to the Gastrointestinal suite.

*Reimbursement and payment process.   There is a well defined commercial code for how payment occurs in most industries.   In health care, each payer creates their own rules.   In aggregate, these rules represent thousands of pages of policies and procedures that a health care provider must follow to be paid.   For example, Medicare's claims processing manual is over 4,000 pages long and this doesn't include national and local coverage determinations, advisories, and other manuals devoted to specific types of Medicare sponsored activities.   Add to this the claims processing rules for Medicaid and private health plans and you have an overwhelming regulatory and compliance challenge.  A cynical person might suggest that payers and government agencies purposely create rules that no provider can possibly follow, then seek compliance penalties for the arcane rules they created.    Providers are in a losing battle to keep up with rules that are in a constant state of flux.

These are all great observations.  

My personal goal is to build software and workflow processes that make the complex seem easy, reducing the burden on providers so that they can focus on what's really important, the patient.   That's why the work for a healthcare CIO will never be done.

Cool Technology of the Week

I've been riding between meetings in the Boston area for 2 years using my Strida folding bike.

It's been great for me but not everyone has a  bike they can carry with them into the office.  

Now, there's a new way to get around Boston - Hubway , funded in part by Beth Israel Deaconess Medical Center.

Using solar powered, cellular connected, high tech bicycle racks with well engineered nearly maintenance free bicycles, it's now possible to commute between 61 stations in the Boston area for a low annual membership or daily fee.

The technology was perfected in Montreal and has solved the problem of bicycle theft, availability, and parking.

It's a truly amazing system and one that is a model for cities around the globe.

A secure, internet connected, solar powered way to rent a bike when you need to ride between points in the Boston area.  That's cool!