Last Friday our practice had an opportunity to practice our disaster recovery protocol. This was actually good news; we are replacing our 6 year old servers with virtual servers and a storage area network (SAN). The implementation plan required more down time than a weekend would provide so we added a Friday to it.
Unlike a real disaster we had the opportunity to prepare immediately before, knowing exactly when the system was going down. We printed our clinic schedules and printed the clinic note from the last visit for patients who had appointments. We did not cancel appointments or reduce the number of patients seen. For appointments requiring other documentation (i.e., pathology reports for post-surgery visits) we printed those. We also printed a generous number of our most common handouts and got our paper prescription pads ready.
The day was free of major problems and the staff performed very well. The experience was very interesting:
- My description above is a little too rosy. We ended up searching for some documents at the last minute. We ran behind schedule and there was a mild degree of disorganization throughout the day.
- Patients were very understanding about the delay and about our EMR being down. Everyone understands that computers go down and I was thankful to see that our patients remembered that. In some cases I had to ask patients to refresh my memory regarding prior visits since I didn’t have the entire chart available. No one seemed to mind.
- I was still able to use my PC and Dragon Speech. Dragon runs locally on my desktop. I dictated notes in Notepad and saved a separate note for each patient. On Monday, my assistant will create Friday’s chart notes in the EMR, enter their data (vital signs etc.) and route the notes to my desktop. Then I will copy / paste my notes into the EMR chart.
- Dictating unstructured chart notes into my PC was refreshing. It was also an impressive reminder of how much garbage CPT forces us to add to our notes. Even for complicated, new patients I was able to record everything relevant in less than two-thirds of a page. Adding the CPT-required material almost doubles the size of the note without adding any relevant, useful data.
- With the system down the front office could focus on patient service without having to obsess over data entry.
- With all the extra paper floating around the back office was a mess. I can’t imagine going back to paper charts.
- We were still compromised operationally when we had a patient who needed to schedule surgery. Without the EMR workflow engine we could not print customized surgical packets. Handwriting surgical consents is not acceptable. We will have to catch up on this workflow next week.
Take home lessons:
We need to improve our disaster readiness, but at the moment our readiness is not too bad. Our new SAN is configured to perform incremental backups every night and full backups every weekend. The virtual servers and SAN will allow us to redirect workload in the event of a server or hard drive failure. For a network failure we can follow the protocol we just rehearsed, but we should take the time to update our hard copies of handouts and surgical consents. This is interesting because on more than one occasion we have had IT vendors try to push their way in to our practice using our presumed lack of disaster readiness as their excuse.
Once our new servers and SAN are settled in I may look into an automated method of copying appointment schedules and recent chart notes for patients with appointments to a local PC in each office rather than wasting all that time and paper every day.
I understand that all that front office data entry is a necessary evil. But I never realized how much it distracts the front office from tending to patients. I am going to double my efforts to make it attractive for patients to enter their own data online in advance of the visit – or at least have them use the patient portal to enter their own data in the waiting room. As much as we have tried so far, our patient portal still has not caught on with our patients in the waiting room, and there has been only moderate use from home. I don’t think our web portal is good enough yet.
The above example is not really disaster recovery, but planned downtime. The problem for most practices is that is their server goes down, there is no copy of the last note or of the schedule to revert back to.
Real Disaster Recovery requires a way to get functional in the case of a server failure or facility problem. Virtualization is the best way to ensure uptime in the case of hardware problems, but it still does not address what happens if the power goes out or a pipe breaks causing flooding.
Preparedness in the event of an outage is essential and – amidst the rush to EHR adoption – it’s often done too late, after a failure. In that regard Dr. Koriwchak’s approach and insights are excellent. So too is the comment from Mr. Bletnitsky regarding the implications of downtime.
But more emphasis should be placed on reducing the risk of downtime in the first case. Call it preventive medicine for EHR. Yes, you need effective back up, recovery and continuity measures, like you need a doctor when you’re sick. But common sense is to try and avoid being sick. Or in this case, reduce the risk up front of downtime. Ironically, healthcare reform is ultimately trying to achieve this, so should the implementation of an EHR system.
Virtualization doesn’t ensure uptime in the case of hardware problems. It efficiently uses resources but still requires an application and associated processes to restart – and if your SQL server is one of those processes that went down you’re looking at hours and not minutes of recovery even if everything goes perfectly. Too often things don’t however.
Ensuring uptime requires three things – resilient technologies, proactive monitoring to identify and mitigate failures before they occur, and best practices. These can be achieved working with companies like Stratus and achieve lower costs with far less IT staff time and keep your EHR system healthy.
Sorry I didn’t read this blog until today Mike, and then only because, while starting my first blog on which students can complete assignments, I ran across you on “Blogger” and that I “follow” your blog. Thought you might find it interesting–scary even?–that we happened to pick the same “watermark” background. I chose it due to my affinity for Richard Bach’s short novel “Jonathan Livingston Seagull”. Hope all’s going well with you and yours.
Henry