The NIST Workshop on EHR Usability

As much as I’d like to visit DC (I’ve never been), I wasn’t able to make it out there to attend the NIST workshop on EHR usability. However, Carl Bergman from EHR Selector did make it to the event and sent the following notes on EHR usability according to NIST.  Most of the speakers name link to their slides in PDF format.

National Institute of Standard and Technology’s Workshop on EHR Usability

This week I went to a NIST workshop examining the state of EHR usability. The workshop was at its administrative headquarters, a large 60s building on its sprawling Gaithersburg, MD campus about 20 miles outside Washington.

You might wonder what NIST is doing in the EHR business? I certainly did. NIST’s mission is to promote commerce and technical innovation including methods to determine, independently, the safety and security of a broad range of technologies including software. (It’s part of the Department of Commerce.) Since WW II, this has involved looking at the human factors involved in operation of every thing from nuclear plants to robotics. Interestingly, it’s not a regulatory agency, such as, the FDA or FCC. NIST’s standards work is through consensus building among manufacturers, consumers, regulators, etc.

The workshop, attended by about 200 persons, had two parts:

•      A review of the state of EHR usability studies by academics, practioners and system administrators and,

•      Introduction of NIST’s draft for a usability standard.

Part I. EHR Usability Today. There were many speakers, here’re the ones that had the most new information for me:

•      Mat Quinn of NIST covered its approach and work with ONC on the issue. Notably, NIST has published several documents in the area such as, NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, (NISTIR 7741) which promotes a user centric approach to design and development.

•      I was really taken by Muhammad Walji’s study using a unified framework for EHR testing. The study compared user experience with the VA’s Vista program and a prototype system. It looked at:

o   What percent of an operation was substantive and what was overhead?

o   How long it took users to reach various performance levels.

o   How much memorization tasks took.

o   How many steps tasks required.

o   Error and recovery occurrence.

o   Time to complete defined tasks.

The study then applied its findings to rework the EHRs’ structure and workflow showing potential time and effort savings.

•      Anjum Chagpar of Toronto’s University Health Network. A human factors manager for this large healthcare network, she discussed the problems of integrating various vendor products into their system and their approach to usability and user satisfaction.

•      Buckminster Fuller famously declared, “I am a verb.” Dr. Lyle Berkowitz may not be a verb, but he is at least a gerund. His presentation swiftly covered several topics from HIMSS’ EHR Usability Task Force to usability definitions to stakeholder roles, and applying metrics to see how much of the problem was the system and how much the user.

•      The VA’s Dr. Jorge Ferrer provided several key references on usability studies.

Part II. NIST’s Proposed Protocol. If the first part took a broad and free ranging approach to usability, NIST’s staff approach was more focused. After an outline of the study’s setting and approach, the study director, Lana Lowery, outlined the protocol’s goal: prevention of unacceptable medical errors. These include errors of both omission and commission, for example:

•      Writing an order for the wrong patient.

•      Prescribing the wrong dosage.

•      Omitted information causing an error.

•      Critical delays in delivery due to system design errors.

•      Errors due to incorrect sequencing of actions.

Next, came examples of EHRs allowing errors. Unfortunately, several of the examples weren’t well thought out. For example, a patient ID error showed two patient records on the screen. One had the first patient’s x-ray, but the second patient’s name. Most likely, this would be a database problem or an x-ray production error not an EHR problem.

Robert Schumacher of User Centric, outlined how the protocol would be tested. For example, review and update of a problem list or replacement of one medication with another. The plan included testing several of ONC’s meaningful use functions that had usability factors.

Part III. Workshop Reactions. The workshop finally broke into two discussion groups: one for the draft protocol and the other on consensus building. In both cases, the discussion quickly went off script. Participants were quick to criticize the staff’s error oriented protocol as too narrow. Why, for example, did the protocol focus on internal EHR processes to the exclusion of workflow generated errors?

I understand NIST has a high interest in eliminating catastrophic errors, but I think there is not enough solid evidence on the kind and extent of the problem. No one discounts the need to prevent catastrophic errors, however, much of the EHR error focus is due to anecdotal reports of computer prescribing errors. From what I read, many of these reports are both old and recycled. Does anyone know the actual extent of major errors?

The FDA has developed several systems for dealing with medical device errors. These now include the software that the devices use. Even if the FDA does not regulate EHRs, it may step up its efforts to record important errors. I’d sure like to know FDA’s findings before I started an effort to shape EHRs.

This is not to say that safety is not important in EHRs, obviously the types of errors that are outlined by the staff are major. However, I think there are three points that are missing in the NIST approach:

•      Design for Success. You can’t design for failure. You have to design for success. The object of EHRs, as with any system, must be to accomplish certain ends. If you loose sight of that, you may not make mistakes, but you also will fail your objective.

•      Risk Analysis. Risk analysis measures the impact on a given population of an action, its potential and costs broadly defined. It also specifies mitigation efforts. I’d be far more comfortable about the protocol if there were a risk analysis behind it.

•      Error Handling. There should be more thought to error handling. For example, when the stall warning alarm goes off on a plane, it doesn’t grab the stick and take control. It’s a warning, just that. Physicians should be warned if they are about to prescribe beyond the recommended dose, but they may have good clinical reason to do it.

NIST put on a worthwhile workshop. My guess is that the draft protocol is not going to survive without modifications that take into account a broader range of usability issues and approaches.

About the author

John Lynn

John Lynn

John Lynn is the Founder of the HealthcareScene.com, a network of leading Healthcare IT resources. The flagship blog, Healthcare IT Today, contains over 13,000 articles with over half of the articles written by John. These EMR and Healthcare IT related articles have been viewed over 20 million times.

John manages Healthcare IT Central, the leading career Health IT job board. He also organizes the first of its kind conference and community focused on healthcare marketing, Healthcare and IT Marketing Conference, and a healthcare IT conference, EXPO.health, focused on practical healthcare IT innovation. John is an advisor to multiple healthcare IT companies. John is highly involved in social media, and in addition to his blogs can be found on Twitter: @techguy.

9 Comments

  • I sure hope Vista was held up as unacceptable with regard to usability and not as a positive example. There is no better way to stifle innovation and usability than to have a government agency take responsibility for it.

  • AXEO,
    It’s a pretty interesting question. I’ll have to think about it. I’ve put your comment in a draft post. Hopefully I can get to it soon. Usability is definitely a challenge to identify and measure.

  • In my experience I have seen EHR usability issues are driven by today’s sales process. Providers choose from 3–4 EHR vendors and choose the one perceived by the providers to be the “easiest.” Unfortunately that drives EHR vendors to produce EHRs that appear at first glance to be “easy.” Only after deploying the choice for six months is the hospital or ambulatory practice in a position to fully understand the real and complex design considerations for usability. By that point however, they are too committed to change vendors.

  • Stephen,

    Oops. They put on a worthwhile workshop.

    It did a good job of reviewing usability issues, even if their proposed approach did not address usability as understood by most who were there.

  • Shane McWhorter,
    I think that analysis is dead on. I’d take it one step further and say that doctors and practices get “woo-ed” by the EMR salesperson and then realize after the fact that it’s not everything the salesperson made it out to be. That’s why the contract is so key. So, you can back out if and when needed.

    Carl,
    I updated the post.

  • John: With regard to your inquiry about whether there are really EHR/EMR/HIT safety problems, rest assured that these issues are there. All you need to do is go to the public FDA database, Manufacturer and User Facility Device Experience (MAUDE). Use the Simple Search option. Enter “Electronic Health Records” or “EHR usability” or enter in various EHR vendor names. Back will come numerous “hits” focused mainly on poor UI design of these systems including placement of key items on displays, annunciation (lack of) of recent patient procedures requiring the attending nurse to have to note when something “new” has been done for her patient, etc. You don’t enter things into this database unless they are connected with adverse events, so the consequences are not usually minor.

    I also was a presenter/attender at the NIST workshop also, and noted a general lack of knowledge regarding the current state of practice in error analysis in the human factors work presented. NIST is simply out to remedy this problem through their usability protocols.

Click here to post a comment
   

Categories