Reflections on the Certification Experience

On Friday, January 21,  2011, Beth Israel Deaconess Medical Center completed the certification of its enterprise EHR technologies via the CCHIT EHR Alternative Certification for Hospitals (EACH) program.   Here's the press release.  As I've written about previously, BIDMC (like many academic health centers) has a combination of built and bought technologies that collectively provide interoperability, clinical functionality, and security.   We demonstrated all our Intersystems Cache-based hospital systems and our Microsoft SQL Server-based business intelligence systems.

The process was rigorous, requiring us to follow over 500 pages of scripts and implementation guides in a single 8 hour demonstration.

The staff at CCHIT were remarkable, educating us about the NIST script requirements, emphasizing the need to prepare, and clarifying aspects of the NIST scripts that were ambiguous or seemed clinically unusual.

NIST did a great job creating test scripts rapidly enough to enable vendors and hospitals to certify systems in time for meaningful use attestation.  However, there are a few oddities in the scripts that are only discoverable during pilots in real hospital and eligible provider clinical settings.   I strongly recommend that for all future certification, NIST pilot their scripts before they are issued for general use.

Major lessons learned include:

*If hospitals apply for complete EHR certification, adding new functionality to fill functional gaps may take months.   Once gaps are filled, I recommend that hospitals devote at least 2 weeks and 5 FTEs to reviewing the scripts, analyzing the best way to show the necessary functionality, and practicing the demonstration.

*The most challenging aspects of certification are interoperability and security.    Interoperability requires  data entry of the necessary information to support the transactions specified in the Standards Final rule: immunization submission, syndromic surveillance, reportable lab, patient summary data import, and patient summary data export.   During certification, each of these data streams will be thoroughly analyzed for compliance with NIST validation tools and/or manually inspected.

*Several of the NIST scripts require data entry that seems clinically unusual.    For example, you must place a CPOE order for Darvocet for pain control, even though Darvocet has been removed from the market by the FDA.    Many of the medications included in the scripts are unusual brand name medications that may not be on a hospital's formulary.   One data set in the Reportable Lab script requires that you send a public health entity information about an infection the patient does NOT have  (Stool Culture with a negative result for Shigella).

*The NIST scripts require that you demonstrate data entry of information that is not normally entered by clinicians in a hospital.   The typical workflow for labs is that they are ordered from an external provider (Quest, Lab Corp) or processed by internal lab systems then inserted into the EHR via an HL7 transaction.   The NIST scripts should be revised to clarify that labs should only be shown, not entered.    Diagnosis and Procedure codes are typically created by Health Information Management after discharge and are sent from a Utilization Review system to the EHR via an HL7 transaction.  The NIST scripts should be revised to clarify that data elements not entered by the clinician should only be shown, not entered.

*The NIST scripts require demonstration of functions that may not be part of standard clinical workflow.   During certification, I wanted to demonstrate live transmission of transactions to the Massachusetts Department of Public Health and Boston Public Health Commission.   Neither of these real public health transmissions were acceptable because the NIST security script requires the demonstration of encryption and hashing in real time.   This is equivalent to not trusting the HTTPS in your browser and requiring browsers to display the actual encryption taking place for every web page retrieved.  The NIST scripts should be revised to enable attestation of the use of FIPS compliant encryption for Stage 1.   Hopefully for Stage 2, there will be enough specificity in transport standards so that the test can be accomplished by submitting data to a NIST specified website that illustrates adherence to the transport standard (NHIN Direct, NHIN Connect etc.)

Here are my detailed observations about the NIST scripts in the order of certification demonstration:

Interoperability
170.302 (k) Submission to immunization registries - This is a great script!  The clinical examples are accurate and are typical of the data elements captured in the real world.   The NIST validator tests real HL7 2.5.1 transactions and the implementation guide is very clear.

170.302 (l) Public health surveillance - This is a good script.   There are no specific examples to enter.   The only issue is that there is no NIST validator for the HL7 2.5.1 generated.   This is because of a mistake in the original Standards and Certification Final rule that specified an implementation guide for Disease Reporting, not Syndromic Surveillance.   A revision to the rule removed the original implementation guide but did not a specify a new one, Hence there is no implementation guide to certify against.   This is not a NIST problem,  but a regulatory problem that should be fixed soon.

170.306 (g) Reportable lab results - This is an odd script.  The examples are all "send out" labs from outside laboratories but the script requires demonstration of the data being entered.   How can you enter data provided by an outside lab?   This script should be revised to require only display of data received from an outside lab that was incorporated into an EHR.    The first sample data set for this script is reasonable - a lead level.   The other samples are more complex than is necessary for demonstration of public health reporting (three instances of the reason for reporting, a corrected result, and reporting of a disease the patient does NOT have - negative result for Shigella).   This is good example of a script that needed to be pilot tested before requiring its use.

170.306 (f) Exchange clinical information and patient summary record - this is a good script but it duplicates 170.306 (d)(1).   The first part of the script requires incorporation of a CCR and a CCD into the EHR in human readable form.   To do this, the appropriate Extensible Style Sheet (XSL)  reference needs to be inserted into the files and the CCD.xsl  and CCR.xsl need to be downloaded and placed in the same directory.   How is a hospital IT department supposed to figure this out? The appropriate style sheets should be included in the script with instructions on how to use them.

For CCD, insert

<?xml-stylesheet type="text/xsl" href="CCD.xsl"?>

For CCR, insert

<?xml-stylesheet type="text/xsl" href="CCR.xsl"?>

as the second line of the files.

The second part of the script is to generate a CCD based on complex data sets which include multiple elements that clinicians do not normally enter (hospital generated diagnosis and procedure codes).  The script should be revised to require only display of data, rather than entry, followed by CCD or CCR generation.  The CCD validator created by NIST uses the HITSP C83 specification which was published before the Standards Final Rule was developed.   HITSP C83 required problem lists to be coded in SNOMED-CT, but the final rule allows ICD-9-CM and SNOMED-CT.  You'll need to read Keith Boone's blog for step by step instructions to create a CCD with ICD-9-CM codes that passes validation.

170.306 (d)(1) Electronic copy of health information - this script should be eliminated because it is a duplication of 170.306(f) with slightly different data sets for generation of the CCD and CCR.

170.306 (i) Calculate and Submit Clinical Quality Measures - the script is fine, but the HITSP document which underlies it contains a few mistakes.  I feel responsible for this because I was the chair of HITSP at the time.   The exclusion and inclusion criteria for VTE-6 are incorrect.   They should be

Inclusion
Patients who received no VTE prophylaxis (HITSP Specification pages 367-370) prior to the VTE diagnostic test order date

Exclusion
Patients with age<18
Patients with length of stay>120 days
Patients enrolled in clinical trial
Patients with comfort measures only documented
Patients with VTE Present on arrival
Patients with reasons for not administering mechanical and pharmacologic prophylaxis
Patients without VTE confirmed by diagnostic testing

You'll also find the term anti-thrombolytic used throughout the document.   There is no such thing as an anti-thrombolytic.   It should be anti-thrombotic.

The HITSP quality measures specification was created before the Standards Final Rule was developed, so although both ICD-9-CM and SNOMED-CT are allowed by the Final Rule, the HITSP specification  defines the quality measures using only SNOMED-CT terminology.   This means that every vendor and hospital has to create their own mappings to ICD-9-CM for all the quality computations.  Here's what we used

Ischemic stroke (ICD-9-CM 433.00-438.99)
Hemorrhagic stroke (ICD-9-CM 430-432.99)
Atrial Fibrillation/Flutter (ICD-9-CM 427.31-427.32)
VTE (ICD-9-CM 453.40-453.42, 453.50-453.52, 453.6, 453.71-453.79, 453.81-453.89, 415.19, 416.2)
Psychiatric diagnoses  (ICD-9-CM 290-319)

I'll also make a controversial statement that several of the quality measures are too complex.   Many of the exclusion criteria are likely to have such a small impact on the measure that they should be eliminated.    (Was the patient  discharged/transferred to a federal healthcare facility?  Better exclude them.  Huh?).  I hope that future quality measures eliminate esoteric exclusionary criteria.

An unintended side effect of the quality measures is that they require changes in software and workflow not specified by Meaningful Use.   For example, in order to report on discharge medications for stroke and VTE patients,  you must implement electronic script writing at discharge to record the discharge medications and associated RxNorm codes for inclusion/exclusion.

Finally. the PQRI XML specification is ambiguous.   The ED measures only require 2 data elements, yet some validators require 6 data elements, with the last 4 set to zero.

Clinical
170.306 (b) Record demographics - Very well written, no issues.

170.306 (h) Advanced directives - Very well written, no issues.

170.302 (f)(1) Vital signs  - Very well written, no issues.

170.302 (f)(2) Body mass index - Very well written, no issues.

170.302 (f)(3) Plot and display growth charts  - Entering the data required for the demonstration is a lengthy process.  Best to revise the script to require only display of data, then graphing.

170.302 (g) Smoking status - The only challenge is that you must adhere to the CDC's smoking codes precisely (1=Current every day smoker, 2=Current some day smoker, etc.) despite the fact that the regulation lists only text values and doesn’t require the CDC numeric recodes.

170.302 (e) Maintain active medication allergy list - Very well written, no issues.

170.302 (j) Medication reconciliation - Very well written, no issues.

170.302 (d) Maintain active medication list  - Very well written, no issues.

170.302 (a) Drug/Drug and Drug/Allergy Interactions - The only challenge is that you must demonstrate how decision support can be disabled for drug/drug and drug/allergy interactions.  I can understand disabling selected drug/drug interactions to reduce alert fatigue, but I cannot think of a clinical reason to disable drug/allergy interactions.

170.302 (b) Drug formulary checks - Very well written, no issues.

170.302 (c) Maintain up-to-date problem list - Very well written, no issues.

170.306 (c) Clinical decision support - Very well written, no issues.

170.306 (a) Computerized provider order entry - The challenge is that the required medications are very odd.   Cefzil (cefprozil) suspension is likely not on most hospital formularies.  Darvocet has been removed from the marketplace by the FDA.   This script needs to be revised to reflect mainstream medications used in live healthcare settings.

170.302 (h) Incorporate lab results - Hospitals are likely to have their own internal laboratories.   The lab system and their EHR may share a common database.   If not, HL7 2.5.1 should be used to transmit data between a external lab systems and the EHR.   This script does not support integrated databases nor HL7 2.5.1 standards   Instead it requires that any structured format be sent from a lab system to the EHR. I'm not sure what policy or technology benefit such a demonstration creates.

170.302 (m) Patient specific education resources - Very well written, no issues.  Hospitals are free to use externally accessible resources such as UptoDate, Healthwise, or LabTestsOnline.org .

170.302 (n) Automate measure calculation  - Very well written, no issues.

170.302 (i) Generate patient lists - I believe that the spirit of the Policy and Standards Committee was to be able to demonstrate a single analytic query on EHR data.   The script goes much further than that, requiring filtering and sorting on problems, medications, and lab values.   I believe the script should be revised to allow a simpler demonstration of business intelligence using EHR data.

170.306 (d)(2) Electronic copy of health information - Very well written, no issues.  There is a great degree of flexibility to create and save discharge communications intended for providers.   There is no test data and no standards conformance testing.

170.306 (e) Electronic copy of discharge information  - Very well written, no issues.  There is a great degree of flexibility to create and save discharge communications intended for patients.   There is no test data and no standards conformance testing.

Security
170.302 (o) Access control - Very well written, no issues.

170.302 (t) Authentication - Very well written, no issues.

170.302 (q) Automatic log-off - Very well written, no issues.

170.302 (p) Emergency access - I'm truly confused by the intent of this test script.   I do not know of any Policy or Standards Committee intent to demonstrate the ability for users (not administrators) to override security controls and obtain access to clinical data.    We demonstrated it successfully, but I am unaware of this being a mainstream or desirable function in an EHR.

170.302 (w) Accounting of disclosures (optional) - We elected not to demonstrate this.  I'm not sure why optional requirements are included in certification.

170.302 (r) Audit log - The audit log script requires the ability to filter and sort the log by numerous criteria.   I am unaware of any Policy or Standards Committee intent to demonstrate advanced analytics on audit logs.

170.302 (s) Integrity - The final three security demonstrations are all very odd.   HIEs use data integrity protections and encryption to ensure data travels from point A to point B without modification.    The script requires demonstration of a test harness, not a live system, because encryption and hashing are invisible, just as HTTPS in your browser is invisible.   All three criteria should be revised to use attestation, not demonstration.
170.302 (u) General encryption - as above
170.302 (v) Encryption when exchanging electronic health information - as above

To restate my major points - CCHIT and the certification process are great.   NIST did a heroic job generating the scripts in a short time.   However, the scripts need to be piloted before they are placed in general use to avoid some of the challenges listed above.   I believe the scripts can be revised to substantially reduce the burden on vendors and hospitals without changing ONC/CMS rule compliance and HIT Policy/Standards Committee intent.

I welcome feedback on your own experiences with certification.   Meaningful Use and its associated processes will be the memories we tell our grandchildren about.