Wednesday, November 27, 2013

Rethinking Certification

As stakeholders in payer, provider, and government communities debate the optimal timing of ICD10, Meaningful Use Stage 2, ACA, and HIPAA Omnibus rule deadlines, it's becoming increasingly clear that many hospitals which attested in 2011 and 2012 will not have their 2014 edition certified software installed, training completed, and workflow re-engineered in time for the Stage 2 attestation deadlines.

Now that we have experience with two stages of Meaningful Use, it's also clear that a three year cycle is needed to ensure safe, high value, well adopted, introduction of new IT functionality.

Part of the problem, as I've discussed previously, is that the certification criteria are overly burdensome and in many circumstances disconnected from the attestation criteria, requiring very prescriptive features that go beyond the intent of Policy Committee and Standards Committee.

How did this happen?  When Meaningful Use Stage 2 regulations were being written, ONC entered a "quiet period" in which smart people wrote regulatory language and certification scripts isolated from the rest of the world to ensure there was no bias introduced.   This was a "waterfall methodology" in which elaborate specifications and a long planning process was followed by an isolated development process resulting in a single huge deliverable with little opportunity to validate the result, pilot the components, or revise/improve the product after the fact.  The flaws in the Stage 2 certification scripts are an artifact of the regulatory process itself. taught us that waterfall approaches are risky.    A better approach would be to create certification scripts using an "agile methodology".   Standards and scripts to test them could be developed outside the regulatory process, with iterative stakeholder feedback, testing of components, and rapid cycle improvement.   The regulatory process could point to the standards which would have accompanying implementation guides and test scripts.   There would be no "quiet period" or isolation in the development of certification scripts.    Such an approach would significantly reduce future certification burdens.

In addition to this, I recommend an even more radical redesign of certification.

We should maintain attestation as a demonstration of performance, but limit certification to rigorous standards adoption and interoperability, not prescriptive functionality.

What do I mean?

The Meaningful Use Workgroup believes decision support should be expanded in the future.   I agree.   Although they are now looking at outcomes that demonstrate the use of decision support, their initial work included recommendations for very prescriptive decision support certification criteria including:

Ability to track CDS triggers
Ability to flag preference-sensitive conditions and provide decision support materials for patients
Check for a maximum dose /weight based calculation
Use of structured SIG standards
Consume external CDS interventions
Use info in systems to support maintenance of lists

In effect, this tells vendors how to enhance clinical decision support features.

Let me use analogy.

Suppose that the government decided USB thumb drives are a good thing.   Not only would they specify a USB 3.0 standard, they would require it is black, rectangular, and weighs 2 ounces.    Such prescriptive requirements would stifle innovation since today's USB drives might be in the shape of a key or even mimic a sushi roll.

When evaluating the success of the US healthcare IT program, Congress tends to focus on interoperability - why are there gaps in DOD/VA data sharing or few seamless transitions of care among inpatient and outpatient facilities?

If certification focused entirely on interoperability, EHRs would be a bit more like USB drives.  They might be big or small, black or red, key shaped or sushi shaped.  However, they'll work with any device you plug them into.

I've spoken with many EHR vendors (to remain unnamed) and all have told me that they created software that will never be used by any clinician but was necessary to check the boxes of certification scripts that make no sense in real world workflows.

If certification required rigorous demonstration of outbound and inbound interoperability with no optionality in the standards (use this standard OR that standard), Congress will be happy, patients will be happy, and vendors will be happy.

Once we come up for air after ICD10, MU2, ACA, and HIPAA, I'll be watching any MU3 planning very closely to ensure we do not again make the same mistakes with certification scripts that are untested or too prescriptive.   Let's all focus on universal adoption of enhanced interoperability as a measure of success.


Dan Haley said...

At athenahealth we couldn't agree more emphatically with this post. We've been beating this particular drum for the better part of a year now (see:, and have been pleased to see the point of view gaining traction. We do not write software that will never see the light of day; we do enable functionality in our cloud-based EHR that arguably serves no purpose but to allow checking of those government boxes. That developer time and effort that could be better spent goes to enable that functionality goes without saying. For us, the most important and crucial component to any MU reform is quite simple: Interoperation is key. "Interoperation," an activity, as opposed to mere "interoperability," which describes a capability that too many vendors are still happy to claim in the abstract while actively impeding it in practice.

Colin Banas MD - VCU Medical Center said...

+1 !!!!

Well stated. They missed it badly this time. There is so much wasted and non-value added effort occurring that it makes my soul ache.....

Too rigid. Too soon. Too many competing priorities and a little elephant in the room called ICD-10.....

Hans Buitendijk said...

I can’t agree more with your assessment and effort to create more sanity in this program. A focus on interoperability, key privacy & security guidelines, and full transparency into quality and cost for all stakeholders will do more to find better ways to improve our healthcare system than the highly prescriptive EHR functional requirements which in the end only covers a fraction of the HIT that is necessary to better support the healthcare system.

Frank Poggio said...

Right on. After working with dozens of vendors taking them thru the Certif Program, the process was thrown together as fast as possible to get ARRA going..damn the details...full speed ahead!
Meanwhile I believe the delay of Stage 3 is wolf in sheep's clothing. In my experience having worked through many 2014 tests with clients - the criteria under 2014 became considerably more complex and extensive as each month passed, test criteria were revised almost on a monthly basis. For example the test for Safety Enhanced Design (170.114g3) started out requesting a simple description of your design/development process, to now submitting detailed test scenarios per NISTIR 7742 standards. The DVT (170.314e1)test now requires WCAG 2.0 evaluations, but did not in the early going. This growing complexity is one reason why far less vendors have passed 2014 than 2011. From a vendor perspective, net result is if you wait for the new set of 2015 test criteria those tests will be even more difficult. My advice, don't wait too long. Remember in the healthcare world regulations only grow...never shrink!
Frank Poggio
The Kelzon Group