When The EMR Goes Down, Doctors Freak Out

Earlier this month, health IT superstar John Halamka, MD, MS posted a story talking about how network downtime within a hospital has changed over the past 10 years or so. I thought I’d share some of it with you, because he makes some interesting points about end user perceptions and sensitivities.

First, he tells the tale of a 2002 network core failure of Beth Israel Deaconess Medical Center, where he serves chief information officer. For two days, he reports, the hospital’s users lost access to all applications, including e-mail, lab results, PACS images and order entry, along with all storage. Or as he puts it, “For two days, the hospital of 2002 became the hospital of 1972.”

He then contrasts that failure with a recent one  (July 25 of this year) in which a storage virtualization appliance at BIDMC failed.  Because the hospital was loathe to risk losing data, he and his team chose a slower path to uptime — reindexing the data — which allowed them to avoid data loss. The bottom line was an outage of a few hours.

This outage was a different ballgame entirely, Halamka says. For example:

* In 2002, staff and doctors weren’t incredibly upset, but this time physicians were angry and frantic, with some noting that they couldn’t take care of patients without EMR access.  Here in 2013, end users expect network access to be like electricity, always there short of an act of God. Worse, though downtime simply isn’t acceptable, but procedures for dealing with it aren’t up to that standard yet, he says.

* Doctors are under an incredible set of regulatory burdens, including but not limited to Meaningful U se, health reform, ICD-10 and the Physician Quality Reporting System. They fear they can’t keep up unless IT functions work perfectly, he observes.

* Technology failures of 2013 are tricky and harder to anticipate. As he notes, back in 2002 servers were physical and storage was physical, but today networks are multi-layered and virtualized. While these things may add capability, they also crank up the complexity of diagnosing system failures, Halamka notes.

Halamka says he learned a lesson from the recent failure:

Expectations are higher, tolerance is lower, and clinician stress is overwhelming. No data was lost, no patient harm occurred, and the entire event lasted a few hours, not a few days. However, it will take months of perfection to regain the trust of my stakeholders.

This story does have one ray of sunshine in it — it demonstrates that increasing numbers of doctors depend completely on their EMR, a state devoutly to be wished for by many health IT leaders. But the price of having doctors throw themselves into EMR use is having them riot when they can’t get to the system. Clearly, hospitals are going to have to find some new way of coping with downtime.

About the author

Anne Zieger

Anne Zieger

Anne Zieger is a healthcare journalist who has written about the industry for 30 years. Her work has appeared in all of the leading healthcare industry publications, and she's served as editor in chief of several healthcare B2B sites.


  • The usage of Utilities and Electricity as an analogy is interesting. This is exactly what ‘cloud’ is all about. The advantage being that along with the Cloud systems comes the power of a vendor that can build in redundancies that a single entity can’t, and access to many more resources.

  • The last paragrgh overstates the importance of EHRs. The decompensation dicussed is not EHR related.

    Drs will decompensate when they can’t get to their old notes or reports, and when they can,t document their care.

    This same decompensation happened when the patient chart was not available or when an official progress note was not available to document care.

  • With some EMR’s you can keep a copy or even the originals of patient records in your pocket. Kind of like the old “plug and play.” One does not always need to “have their head in the clouds.”

  • This is exactly the issue with a SAAS or “cloud” based EHR.

    Most/all SAAS EHR vendors will state that you are “required” to have at least 2 methods of accessing the internet in order to have the proper redundancies.

    Few offices actually do this (this should also be part of your contingency operations plan…yes a HIPAA requirement).

    On an “in-house” server system, your dependency on the internet is greatly reduced as you can at least function internally until outside access returns.

    Either way, you need to understand how you are going to function when you system is down (again…this should also be part of your contingency operations plan…yes a HIPAA requirement).

    Think about how much chaos you see at a restaurant when their POS goes down, and that is a simple restaurant.

  • “For two days, the hospital of 2002 became the hospital of 1972.”

    The real question is, who in 2013 …

    is still using staff, procedures, equipment, platforms, approaches, methodologies, technologies, and so on from

    … 1972 ?

Click here to post a comment