Tuesday, January 25, 2011

Reflections on the Certification Experience

On Friday, January 21,  2011, Beth Israel Deaconess Medical Center completed the certification of its enterprise EHR technologies via the CCHIT EHR Alternative Certification for Hospitals (EACH) program.   Here's the press release.  As I've written about previously, BIDMC (like many academic health centers) has a combination of built and bought technologies that collectively provide interoperability, clinical functionality, and security.   We demonstrated all our Intersystems Cache-based hospital systems and our Microsoft SQL Server-based business intelligence systems.

The process was rigorous, requiring us to follow over 500 pages of scripts and implementation guides in a single 8 hour demonstration.

The staff at CCHIT were remarkable, educating us about the NIST script requirements, emphasizing the need to prepare, and clarifying aspects of the NIST scripts that were ambiguous or seemed clinically unusual.

NIST did a great job creating test scripts rapidly enough to enable vendors and hospitals to certify systems in time for meaningful use attestation.  However, there are a few oddities in the scripts that are only discoverable during pilots in real hospital and eligible provider clinical settings.   I strongly recommend that for all future certification, NIST pilot their scripts before they are issued for general use.

Major lessons learned include:

*If hospitals apply for complete EHR certification, adding new functionality to fill functional gaps may take months.   Once gaps are filled, I recommend that hospitals devote at least 2 weeks and 5 FTEs to reviewing the scripts, analyzing the best way to show the necessary functionality, and practicing the demonstration.

*The most challenging aspects of certification are interoperability and security.    Interoperability requires  data entry of the necessary information to support the transactions specified in the Standards Final rule: immunization submission, syndromic surveillance, reportable lab, patient summary data import, and patient summary data export.   During certification, each of these data streams will be thoroughly analyzed for compliance with NIST validation tools and/or manually inspected.

*Several of the NIST scripts require data entry that seems clinically unusual.    For example, you must place a CPOE order for Darvocet for pain control, even though Darvocet has been removed from the market by the FDA.    Many of the medications included in the scripts are unusual brand name medications that may not be on a hospital's formulary.   One data set in the Reportable Lab script requires that you send a public health entity information about an infection the patient does NOT have  (Stool Culture with a negative result for Shigella).

*The NIST scripts require that you demonstrate data entry of information that is not normally entered by clinicians in a hospital.   The typical workflow for labs is that they are ordered from an external provider (Quest, Lab Corp) or processed by internal lab systems then inserted into the EHR via an HL7 transaction.   The NIST scripts should be revised to clarify that labs should only be shown, not entered.    Diagnosis and Procedure codes are typically created by Health Information Management after discharge and are sent from a Utilization Review system to the EHR via an HL7 transaction.  The NIST scripts should be revised to clarify that data elements not entered by the clinician should only be shown, not entered.

*The NIST scripts require demonstration of functions that may not be part of standard clinical workflow.   During certification, I wanted to demonstrate live transmission of transactions to the Massachusetts Department of Public Health and Boston Public Health Commission.   Neither of these real public health transmissions were acceptable because the NIST security script requires the demonstration of encryption and hashing in real time.   This is equivalent to not trusting the HTTPS in your browser and requiring browsers to display the actual encryption taking place for every web page retrieved.  The NIST scripts should be revised to enable attestation of the use of FIPS compliant encryption for Stage 1.   Hopefully for Stage 2, there will be enough specificity in transport standards so that the test can be accomplished by submitting data to a NIST specified website that illustrates adherence to the transport standard (NHIN Direct, NHIN Connect etc.)

Here are my detailed observations about the NIST scripts in the order of certification demonstration:

170.302 (k) Submission to immunization registries - This is a great script!  The clinical examples are accurate and are typical of the data elements captured in the real world.   The NIST validator tests real HL7 2.5.1 transactions and the implementation guide is very clear.

170.302 (l) Public health surveillance - This is a good script.   There are no specific examples to enter.   The only issue is that there is no NIST validator for the HL7 2.5.1 generated.   This is because of a mistake in the original Standards and Certification Final rule that specified an implementation guide for Disease Reporting, not Syndromic Surveillance.   A revision to the rule removed the original implementation guide but did not a specify a new one, Hence there is no implementation guide to certify against.   This is not a NIST problem,  but a regulatory problem that should be fixed soon.

170.306 (g) Reportable lab results - This is an odd script.  The examples are all "send out" labs from outside laboratories but the script requires demonstration of the data being entered.   How can you enter data provided by an outside lab?   This script should be revised to require only display of data received from an outside lab that was incorporated into an EHR.    The first sample data set for this script is reasonable - a lead level.   The other samples are more complex than is necessary for demonstration of public health reporting (three instances of the reason for reporting, a corrected result, and reporting of a disease the patient does NOT have - negative result for Shigella).   This is good example of a script that needed to be pilot tested before requiring its use.

170.306 (f) Exchange clinical information and patient summary record - this is a good script but it duplicates 170.306 (d)(1).   The first part of the script requires incorporation of a CCR and a CCD into the EHR in human readable form.   To do this, the appropriate Extensible Style Sheet (XSL)  reference needs to be inserted into the files and the CCD.xsl  and CCR.xsl need to be downloaded and placed in the same directory.   How is a hospital IT department supposed to figure this out? The appropriate style sheets should be included in the script with instructions on how to use them.

For CCD, insert
<?xml-stylesheet type="text/xsl" href="CCD.xsl"?>
For CCR, insert
<?xml-stylesheet type="text/xsl" href="CCR.xsl"?>
as the second line of the files.

The second part of the script is to generate a CCD based on complex data sets which include multiple elements that clinicians do not normally enter (hospital generated diagnosis and procedure codes).  The script should be revised to require only display of data, rather than entry, followed by CCD or CCR generation.  The CCD validator created by NIST uses the HITSP C83 specification which was published before the Standards Final Rule was developed.   HITSP C83 required problem lists to be coded in SNOMED-CT, but the final rule allows ICD-9-CM and SNOMED-CT.  You'll need to read Keith Boone's blog for step by step instructions to create a CCD with ICD-9-CM codes that passes validation.

170.306 (d)(1) Electronic copy of health information - this script should be eliminated because it is a duplication of 170.306(f) with slightly different data sets for generation of the CCD and CCR.

170.306 (i) Calculate and Submit Clinical Quality Measures - the script is fine, but the HITSP document which underlies it contains a few mistakes.  I feel responsible for this because I was the chair of HITSP at the time.   The exclusion and inclusion criteria for VTE-6 are incorrect.   They should be

Patients who received no VTE prophylaxis (HITSP Specification pages 367-370) prior to the VTE diagnostic test order date

Patients with age<18
Patients with length of stay>120 days
Patients enrolled in clinical trial
Patients with comfort measures only documented
Patients with VTE Present on arrival
Patients with reasons for not administering mechanical and pharmacologic prophylaxis
Patients without VTE confirmed by diagnostic testing

You'll also find the term anti-thrombolytic used throughout the document.   There is no such thing as an anti-thrombolytic.   It should be anti-thrombotic.

The HITSP quality measures specification was created before the Standards Final Rule was developed, so although both ICD-9-CM and SNOMED-CT are allowed by the Final Rule, the HITSP specification  defines the quality measures using only SNOMED-CT terminology.   This means that every vendor and hospital has to create their own mappings to ICD-9-CM for all the quality computations.  Here's what we used

Ischemic stroke (ICD-9-CM 433.00-438.99)
Hemorrhagic stroke (ICD-9-CM 430-432.99)
Atrial Fibrillation/Flutter (ICD-9-CM 427.31-427.32)
VTE (ICD-9-CM 453.40-453.42, 453.50-453.52, 453.6, 453.71-453.79, 453.81-453.89, 415.19, 416.2)
Psychiatric diagnoses  (ICD-9-CM 290-319)

I'll also make a controversial statement that several of the quality measures are too complex.   Many of the exclusion criteria are likely to have such a small impact on the measure that they should be eliminated.    (Was the patient  discharged/transferred to a federal healthcare facility?  Better exclude them.  Huh?).  I hope that future quality measures eliminate esoteric exclusionary criteria.

An unintended side effect of the quality measures is that they require changes in software and workflow not specified by Meaningful Use.   For example, in order to report on discharge medications for stroke and VTE patients,  you must implement electronic script writing at discharge to record the discharge medications and associated RxNorm codes for inclusion/exclusion.

Finally. the PQRI XML specification is ambiguous.   The ED measures only require 2 data elements, yet some validators require 6 data elements, with the last 4 set to zero.

170.306 (b) Record demographics - Very well written, no issues.

170.306 (h) Advanced directives - Very well written, no issues.

170.302 (f)(1) Vital signs  - Very well written, no issues.

170.302 (f)(2) Body mass index - Very well written, no issues.

170.302 (f)(3) Plot and display growth charts  - Entering the data required for the demonstration is a lengthy process.  Best to revise the script to require only display of data, then graphing.

170.302 (g) Smoking status - The only challenge is that you must adhere to the CDC's smoking codes precisely (1=Current every day smoker, 2=Current some day smoker, etc.) despite the fact that the regulation lists only text values and doesn’t require the CDC numeric recodes.

170.302 (e) Maintain active medication allergy list - Very well written, no issues.

170.302 (j) Medication reconciliation - Very well written, no issues.

170.302 (d) Maintain active medication list  - Very well written, no issues.

170.302 (a) Drug/Drug and Drug/Allergy Interactions - The only challenge is that you must demonstrate how decision support can be disabled for drug/drug and drug/allergy interactions.  I can understand disabling selected drug/drug interactions to reduce alert fatigue, but I cannot think of a clinical reason to disable drug/allergy interactions.

170.302 (b) Drug formulary checks - Very well written, no issues.

170.302 (c) Maintain up-to-date problem list - Very well written, no issues.

170.306 (c) Clinical decision support - Very well written, no issues.

170.306 (a) Computerized provider order entry - The challenge is that the required medications are very odd.   Cefzil (cefprozil) suspension is likely not on most hospital formularies.  Darvocet has been removed from the marketplace by the FDA.   This script needs to be revised to reflect mainstream medications used in live healthcare settings.

170.302 (h) Incorporate lab results - Hospitals are likely to have their own internal laboratories.   The lab system and their EHR may share a common database.   If not, HL7 2.5.1 should be used to transmit data between a external lab systems and the EHR.   This script does not support integrated databases nor HL7 2.5.1 standards   Instead it requires that any structured format be sent from a lab system to the EHR. I'm not sure what policy or technology benefit such a demonstration creates.

170.302 (m) Patient specific education resources - Very well written, no issues.  Hospitals are free to use externally accessible resources such as UptoDate, Healthwise, or LabTestsOnline.org .

170.302 (n) Automate measure calculation  - Very well written, no issues.

170.302 (i) Generate patient lists - I believe that the spirit of the Policy and Standards Committee was to be able to demonstrate a single analytic query on EHR data.   The script goes much further than that, requiring filtering and sorting on problems, medications, and lab values.   I believe the script should be revised to allow a simpler demonstration of business intelligence using EHR data.

170.306 (d)(2) Electronic copy of health information - Very well written, no issues.  There is a great degree of flexibility to create and save discharge communications intended for providers.   There is no test data and no standards conformance testing.

170.306 (e) Electronic copy of discharge information  - Very well written, no issues.  There is a great degree of flexibility to create and save discharge communications intended for patients.   There is no test data and no standards conformance testing.

170.302 (o) Access control - Very well written, no issues.

170.302 (t) Authentication - Very well written, no issues.

170.302 (q) Automatic log-off - Very well written, no issues.

170.302 (p) Emergency access - I'm truly confused by the intent of this test script.   I do not know of any Policy or Standards Committee intent to demonstrate the ability for users (not administrators) to override security controls and obtain access to clinical data.    We demonstrated it successfully, but I am unaware of this being a mainstream or desirable function in an EHR.

170.302 (w) Accounting of disclosures (optional) - We elected not to demonstrate this.  I'm not sure why optional requirements are included in certification.

170.302 (r) Audit log - The audit log script requires the ability to filter and sort the log by numerous criteria.   I am unaware of any Policy or Standards Committee intent to demonstrate advanced analytics on audit logs.

170.302 (s) Integrity - The final three security demonstrations are all very odd.   HIEs use data integrity protections and encryption to ensure data travels from point A to point B without modification.    The script requires demonstration of a test harness, not a live system, because encryption and hashing are invisible, just as HTTPS in your browser is invisible.   All three criteria should be revised to use attestation, not demonstration.
170.302 (u) General encryption - as above
170.302 (v) Encryption when exchanging electronic health information - as above

To restate my major points - CCHIT and the certification process are great.   NIST did a heroic job generating the scripts in a short time.   However, the scripts need to be piloted before they are placed in general use to avoid some of the challenges listed above.   I believe the scripts can be revised to substantially reduce the burden on vendors and hospitals without changing ONC/CMS rule compliance and HIT Policy/Standards Committee intent.

I welcome feedback on your own experiences with certification.   Meaningful Use and its associated processes will be the memories we tell our grandchildren about.


Jenni P said...

Regarding 170.306 (g) Reportable lab results, I believe NIST should only require displaying of the lab result for any test. I believe the validation of the lab result being from a certified EHR (or EHR module) should be a prerequisite; not part of this test itself. My reasoning is that a 3rd party, such as an HIE, may be looking for certification as an EHR module for ELR. They 'consume' the lab result and would generate the ELR (2.5.1) message. I have submitted this to ONC and NIST. NIST said they would revisit this portion of the test once ONC confirms this. There are several phrases in the final rules and FAQs that lead me to believe ONC will confirm that an HIE can become a certified EHR module for its clinical stakeholders.

Anonymous said...

I feel you hit the nail on the head with many of these comments. I'm from an EHR vendor and I directly participated in our certification prep and demonstration. We struggled with many of the same concerns/issues as you and I hope that these changes are made before round two.

Terry K said...

John, We recently took our ambulatory EHR through certification with CCHIT and ran into a number of the oddities you identified, particularly in the realm of unusual medications. I too hope that NIST brings their scripts up to date prior to round 2 as there is more than enough to concern ourselves with during testing beyond how we identify a medication the FDA has pulled from the market. I would also echo your compliments towards CCHIT. We found their organization to be extremely helpful and supportative throughout the certification process.

John Phelan said...

great Post John. All of us are greatful for your continued leadership and help. How have you addressed the patient didgital health data delivery post care?

Anonymous said...

what tool have you used to validate 170.306 (i) Calculate and Submit Clinical Quality Measures.

How is CCHIT testing if the file created is in the correct format?

John Halamka said...

That is the subject of my blog on January 31!

Anonymous said...

Hi John,

I would like to know what NIST means 'Message type' in public health surveillance. Vendor need to provide the message type in order to verify the conformance of the message.
Could you please specify what message type is? It would be very helpful if you direct me to some of the known message types.

John Halamka said...

There is no implementation guide or validator of the Public Surveillance transaction. During certification you have the option of specifying HL7 2.3.1 or HL7 2.5.1. The HL7 version is the message type. For a sample, see my February 1, 2011 blog post.

Anonymous said...

I have a comment about the quality measures. HITSP measures have not been updated since April-2010 and have glaring errors. CMS/TJC specs for the same measures are up to date and have much better explanation. My recommendation is to use CMS measures instead of HITSP. They also come with complete list of ICD codes so there is no need to come up with a mapping from SNOMED to ICD.

Any thoughts???

John Halamka said...

My understanding is that the Arizona QIO is updating the HITSP measures.

You can check the versions published yesterday (http://www.qualityforum.org/Projects/e-g/eMeasures/Electronic_Quality_Measures.aspx?section=PublicandMemberComment2011-02-012011-04-01#t=1&s=&p=) to see the difference for some of the newly retooled measures: inpatient includes 16 – a few are 0147, 0527, 0528, 0529, 0300, 0163, 0164.

John Halamka said...

Also, here's a response from CMS:

"Thank you for pointing out some of the discrepancies in the HITSP specifications. The current hospital/CAH electronic specifications posted are considered final and should be used for EHR Incentive Program reporting. We are aware of needed updates to the current version of the HITSP specifications and we are actively working to post supplemental specifications on the CMS website in the future to assist implementers with technical updates and requested guidance."

Unknown said...

Hi Halamka,

Your post was very useful.

I have few queries regarding modular certification( CQM objective ).

1 )For a modular certification do we / don’t we need to demo the working solution on top of a transactional system (EHR) application(s)?

2)For modular certification, Do we have to enter NIST sample data into an EHR, then show the data is flowing from the EHR to my CQM module or can we show the data flowing from a staging database without integration to a specific EHR to supply data (NIST data entered by a tester through a simple UI to populate the staging database).

3)If i get a modular certification for CQM objective alone, can i say my CQM module will work on top of any EHR or is it means it will work on top of a particular EHR product(which i used as a data source for my modular certification) only ?

Thanks for your time

Ivan said...

Hi, John,

As usual, very interesting info.

Our organization has recently been researching InterSystems technologies and here I see you mentioning that your EHR is built upon them. I would really appreciate if you could spare a few words on the topic.

Are you using only Cache for storage or CSP as well? Are you using Ensemble?

Why did you decide to go with Cache and not more traditional MSSQL or Oracle DBMS?

Does InterSystems group of technologies have much competition if you are building an EMR system?


Anonymous said...

Hello John,

Very useful blog. Looking at your review for 170.306 (f) , you said we only need to be able to display the imported CCD in a human readable format. Is it not a requirement to add the corresponding data elements (procedures,medications,lab results,diagnosis.) as structured data in the patient's account?

Thank you...

John Halamka said...

Correct, no import as structured data is required.

Anonymous said...

Hi John,

Regarding 170.306 (f), you suggested using CCD.xsl to fulfill the "human readable format" requirements for HL7-CCD. The XSL seems to look at the / element, which is just a free text not subjected to any rules from NIST. Shouldn't the XSL look at the / segment instead? In other words I can very well have an empty / element in a valid CCD. This would however fail the "human readable" requirement.
Please let us know your thoughts.

Much thanks!!

John Halamka said...

During certification, you are provided with a CCD containing all the necessary data elements to display. These are checked for accuracy during the human readable display demonstration.

Anonymous said...

Hello John,

Very useful blog...

Per your comment - "170.302 (i) Generate patient lists - I believe that the spirit of the Policy and Standards Committee was to be able to demonstrate a single analytic query on EHR data."
Will canned reports (1 report per scenario listed in the test script) suffice the requirement instead of using a single analytical query on the EHR data? This means the user will just click on the report link to see the results.

Thank you.

Anonymous said...

Hello John,

Very nice blog!!!
A question on the HL7 CCD. For the INR lab test, as the units are not present, how would the CCD observation block look like? Below is a snippet using NONE as the unit:

0.9-1.1 NONE

Much Thanks!!!

J said...

Can you elaborate on how you were able to demonstrate compliance with the scripts regarding the following?

Emergency Access - Did you customize the software? From how I'm reading the requirement, we grant a set of users emergency access, and they can only use it if they click an option available to them to invoke the emergency access. This is a horrible requirement!

Integrity & Encryption - How did you demonstrate this within or outside the software? Did you do this simply with data that was exported to a file (CSV, PDF)?

Anonymous said...

regarding 170.306 (d) 1: i am part of a team working on taking the patient health information via a Patient Portal. As a result, we are using certified EHR software to do so. However, to get the CCD for 170.306(d)1, we would like to use the image stored in our imaging system rather than pulling it from the EHR data store. is this even an option since we clearly would not have it in the HL7 CCD format?