A Failed Hard Drive on Your EMR Server

Note: This post starts a bit technical, but even those less technical should benefit from the second half of the post.

Today I had the very exciting event of a hard drive dieing on one of my servers. Luckily, I had the drives mirrored and nobody noticed. The server just kept running off of the good drive. I’ve had this happen a couple times and it always makes me happy to say that we lost a hard drive and nobody noticed the difference. Nice to know that things work the way they should be working.

However, the thought of losing the other hard drive before I could restore the second drive is always a little bit tense. Luckily for me, I was able to just connect to the raid controller (yes I just lost half my readers) and rebuild the failed hard drive and we’re back in business (at least for now).

I admit that when this happened I went straight to my backups to make sure that they were working correctly and I immediately made a backup of all the databases on that server. Of course, the thought being that if I lose both hard drives I’d at least have my backups (which I’d saved to another machine).

Enough technical already. My point of this post was that this event really made me think about what a doctor’s office would do if they lost a hard drive. They better hope they have good IT staff to be able to fix it. It’s far from rocket science, but you do have to be careful in these situations. The other important lesson is who do you have monitoring that the drive failed? Most servers have the ability to notify you when a drive fails. However, how many doctors or IT people that support doctors have set up these notifications?

I wish I could say this was an isolated incident that’s rare. Over the past 4+ years in my current job, I’ve had 3 or 4 hard drives fail. Not a huge number, but also enough to require a good plan for when this happens. I’ve often heard that the first thing to die on a server is the hard drive. Why? Because the hard drive is a moving part (the disk spins) and moving parts break. So, you just have to expect for these things to happen and plan for it.

At the end of the day, nothing was lost and there was no down time for our EMR server. That’s the good news about a well designed EMR server. The problem comes when no one’s watching for these events or when the EMR server isn’t installed in order to plan for these types of events.

I’m pretty sure my SaaS EMR friends are going to love this post. There’s no doubt this is a huge plus for SaaS (hosted) EMR companies, but there are other challenges associated with SaaS that should be considered as well. I’ll let the SaaS enjoy this benefit for now and leave those comments for another post.

About the author

John Lynn

John Lynn is the Founder of HealthcareScene.com, a network of leading Healthcare IT resources. The flagship blog, Healthcare IT Today, contains over 13,000 articles with over half of the articles written by John. These EMR and Healthcare IT related articles have been viewed over 20 million times.

John manages Healthcare IT Central, the leading career Health IT job board. He also organizes the first of its kind conference and community focused on healthcare marketing, Healthcare and IT Marketing Conference, and a healthcare IT conference, EXPO.health, focused on practical healthcare IT innovation. John is an advisor to multiple healthcare IT companies. John is highly involved in social media, and in addition to his blogs can be found on Twitter: @techguy.

12 Comments

  • Hi,
    I am new to the field of EMR and am presently developing a suite of Open EMR .
    Your blog has been extremely beneficial in this journey.
    My question has nothing to do with your post so pllease pardon the piggyback…
    Once my software is ready, where do I get it certified that it is indeed HIPAA compliant? Is it the CCHIT that does this or some other federal body?
    I would appreciate any answer you could give me.
    Best Regards,
    Heeru Chari

  • @Heeru,
    Just a quick note, as I don’t want to pull the comment thread away from the topic any further: Software cannot be HIPAA compliant. Software canot be HIPAA certified. A dataset can be certified as deidentified according to HIPAA guidelines, and a company’s portability, and accountability practices can be labeled compliant.
    Hope this clears it up a bit.
    That said, it may be beneficial to do this research before doing further development, as information gathered along the way will likely influence your product roadmap.
    Best of luck with the new venture,
    David

  • I was going to remove the comment and reply, but since we started down this path, might as well leave it.

    It’s definitely a problem with HIPAA compliance. EMR vendors just say there software is HIPAA compliant or not. It’s up to the doctor to see if indeed the software has met the HIPAA requirements of privacy, disclosure, backups, etc or not. CCHIT definitely does not do this and no one else would because of liability I presume.

    At the end of the day, no one will really know if it’s HIPAA compliant or not until it goes to court. Even then, a doctors personal practices can make it not HIPAA compliant because they don’t have the proper safeguards.

    I hope this helps.

  • I don’t know why I stopped reading this blog – will have to start up again!

    I’m glad you have tier 1 (redundancy) and tier 2 (backups) in place. A little comment however:
    “That’s the good news about a well designed EMR server” Actually, based off the failure rate, it sounds like you’re using SATA drives, which are not designed for 24/7 use, like in your EMR database server. A well designed server would take advantage of the speed and longevity of SAS hard drives. Since I started my company 3 years ago (healthcare specific IT for ambulatory practices), I’ve implements 100’s of SAS drives with 0% failure rate. In addition, when buying servers, buy HP or DELL, these brand names have the tools built in to monitor the hardware and can alert your IT resource (if properly setup). In addition, the IT resource should deploy a 3rd party tool such as Kaseya or Hound Dog to help monitor all aspects of the server and network. We suggest utilizing a backup system that can provide business continuity and disaster recovery, coupled with offsite (online) backup, which would be our tier 3 solution. Also, don’t forget the bad hard drive needs a low-level format to get rid of all that patient data!!

  • I don’t know why you stopped either TrentP. I’ll forgive you if you start again.

    I didn’t mention it in this post. Sounds like a good post for another day. I do have SQL Server replication happening. Plus, I’m in a unique environment where I replicate to a server about a mile away. Then, we do have offsite backups as well. We’ve covered backups, business continuity and disaster recovery pretty well.

    I didn’t bring most of those up, because most doctors offices can’t afford a robust backup system like I’ve implemented in my day job.

    Also, there actually SCSI drives because the server is about 4 years old and about to be replaced. SATA was the only other option at the time. I’m about to reorder and I’m considering the SAS option. So many things to keep up with, which goes back to how are doctors suppose to know this stuff.

    It is a Dell server with built in notifications, but my point in the post above is how many doctors offices actually turn on these notifications. Goes back to a similar discussion of “Alert Fatigue” with clinical decision support. Same thing can happen with these server notifications. Although, Dell has been pretty good in that area.

    Glad to have you back as a reader.

  • Well, you’re officially back in my RSS feed – I’ll try to post more often (hopefully less criticizing!). Sounds like you have a good backup system in place. I agree that cost is very much a factor in technology decisions, it’s very important to have that trusted advisor that can explain the features and benefits of the technology. Unfortunately, when shopping EMR, many docs (or office managers) do not care about the benefits of SAS vs SATA hard drive or the like, only looking to see if the bottom line fits their budgets and trusting your word best solution. Also, if the docs have it in their budget, I would definitely sign up for monitoring and maintenance program with your IT resource to get the most out of your technology investment. Some docs want to be invloved with the tech, but others just want it to work. Anyway, you want fast hard drives, look in to solid state drives (no moving parts)!

    Can you send me your private email John, I have a few questions for you?! -TP

  • I don’t mind criticizing as long as it’s criticizing the comments and not me. I like to hear dissenting opinions. You learn more from those than you do from those who just agree with you.

    Solid state drives are really nice, but a bit pricey. I don’t mind if a drive fails too much since I have 4 hour support if/when it does fail. We’ll see if prices have come down for the solid state since I last looked.

  • Greetings again,

    We have a nice 15-drive hot swappable array for our EMR database (RAID 10). We’ve had ~three drives die in the life of our server, and all we had to do is yank the bad drive, pop in a fresh drive, and the magic of RAID does the rest!

    Of course, all that gear cost $$$, plus it took a guy who knew what he was doing to spec it out. If an organization refuses to have in-house IT, they should really look at some kind of hosting service.

    As for the in-house vs. outsourced debate, there’s (at least) three types of docs I can identify:

    * The ones who just want to use the tool, as it comes
    * The more technical ones that want to get involved
    * The ones who DON’T want to get involved, but who want it THEIR way and aren’t going to bicker with/wait for some outside organization to set it up their way, thank you very much

  • Steven,
    Thanks for illustrating my point. While I get excited thinking about 15-drive hot swappable arrays in a RAID 10, I’m pretty sure that most doctors started rolling their eyes.

    You’re right about the magic of RAID. However, even RAIDS can have troubles depending on the controller, depending on the size/type of drives. Not to mention knowing that it needs to be swapped.

    All of this means that I’ll always have a job.

  • Dwayne Carter

    I constantly fight with my clients over two things. UPS and Backup Hardware and Software. It always amazes me…that these critical area’s are the first to get “cut” during the spec’ing phase.

    I’m just dumbfounded.

  • But Dwayne, the in house server is so much more secure and reliable than cloud health IT. Yes, there’s a lot of things like this that often get cut because of budget or just plain ignorance.

Click here to post a comment
   

Categories