EMR Data Archiving

In my previous post I talked about a few of the options related to shredding or archiving your paper charts after you’ve implemented an EMR and the paper charts aren’t being accessed.

Now let’s take a look at what’s just begun to come up in our clinic: EMR Data Archiving

Archiving Old EMR Records
Once you’ve been on an EMR for a number of years, you start to think about all of the data that’s stored in your EMR. Do you really want to store ALL of the information you’ve entered into an EMR in perpetuity? The answer as usual is maybe.

Quick Disclaimer: Before you do any EMR data archiving, you better talk to a good lawyer to make sure what you’re doing is legal in your state. I am not a lawyer and don’t even play one on TV. Just don’t be surprised if your lawyer say there’s no case law on this and so they’ll only be able to give you a best guess.

Now back to the point. The concepts in this are very similar to retention of paper charts. In fact, I’m guessing the laws for this will be similar to the laws on retention of paper charts.

On the one hand, it’s a great thing to be able to have years and years of a patient’s history available to you at the click of the button. In fact, in certain cases where you’re dealing with chronic conditions having all of the EMR data forever could be VERY valuable.

However, I see two possible problems with retaining all of your EMR data forever. First, there is a potential liability issue if you retain all of those records which are very old. If you’ve disposed of the data properly and according to state and federal laws, then you are no longer liable for the removed information that’s stored in the EMR. The second issue is that your EMR database grows larger and larger each year you use your EMR. Between all of the data that’s entered in the EMR and all the various scanned documents that are added to the EMR, the size of the database can be a huge issue after a number of years.

I’ll leave the first problem of liability to the lawyers. The second problem is really interesting. As your database grows, now database backups go from 5 minutes to sometimes hours (in some cases). Restoring all of that data will take a long time as well. At some point your server can run out of disk space for your database and will you even notice that it’s out of space before it’s too late and your server is in trouble? These are just a few of the issues related to storing all of your EMR data forever.

I don’t know of any EMR company that has implemented a solution for this problem. Although, it’s theoretically possible that you could extract the data from your EMR that was previous to a certain date. There could be multiple options with that data. For those concerned about liability, they could delete the extracted data and it would be gone forever. For those just concerned with the database size, you could move the extracted data onto another device or even just saved in a safe location (probably more than one location). In fact, you could even add it as its own database and your EMR company could give you a way to switch to the “archived EMR database” to see all of the archived data. Doing this will make your EMR database much smaller and much more manageable.

I have a feeling this post might have gotten a little too technical. Oh well, I guess sometimes the tech guy in me just comes out.

Also, lest someone starts typing a comment that the database will never get that big to matter. Be careful in saying never. Between scanned image files which are getting bigger and bigger as the quality improves and just the sheer length of time that people have owned an EMR this can happen. In fact, it has happened.

About the author

John Lynn

John Lynn

John Lynn is the Founder of the HealthcareScene.com, a network of leading Healthcare IT resources. The flagship blog, Healthcare IT Today, contains over 13,000 articles with over half of the articles written by John. These EMR and Healthcare IT related articles have been viewed over 20 million times.

John manages Healthcare IT Central, the leading career Health IT job board. He also organizes the first of its kind conference and community focused on healthcare marketing, Healthcare and IT Marketing Conference, and a healthcare IT conference, EXPO.health, focused on practical healthcare IT innovation. John is an advisor to multiple healthcare IT companies. John is highly involved in social media, and in addition to his blogs can be found on Twitter: @techguy.


  • Your comments on EMR Data Archieving only is the tip of the ice burg on benefits and challenges. Managing data has always been and always will be a challenge; especially because no one knows what will be needed in the future. 5 years from now a study group may have a value in the data and would be willing to pay for that use. that needs to be balanced against the govenment policies and guidelines; as you sggested.

    Has any vendor given any thought to storing data in the cloud? Would there be any value in this type of approach? What would be the pros and cons of such an approach.

  • Rick,
    Certainly there are cases where people might want to do a clinical study 5 years down the road. However, I know of very few doctors who do these types of studies.

    Although, the point is interesting. Are we getting rid of all the data that could be very useful. Maybe when the data is archived it could be de-identified and maintained for research purposes only.

    No doubt, there’s a lot of factors involved in archiving the old data. I don’t think there’s one right answer, but that’s why I wanted to start the discussion.

  • John
    A quick search revealed the apparent laws in the UK for data retention of medical records.

    I am sure similar laws apply in the USA (note the reference to keeping records 10 years of death of the patient).

    I don’t see this problem of database growth as different from other industries. What rules does the FASB or SEC have for maintaining financial records?
    What is more cumbersome to manage, dozens or hundreds of steel cabinets containing paper, or dozens or hundreds of mega/terabytes of disk space. I’d go for disk anytime. I know that the IRS requires me to keep five years of tax returns. Using electronic Tax software I can keep store that on a CD, or five boxes in the garage. Take your pick.

  • Hi Jonathan,
    The difference between the financial stuff and an EMR is the number of papers you often have to scan for your EMR. Sure, they have documents in the financial world too, but they’re usually in some document management software instead of fully integrated in your EMR.

    There’s a lot of interesting debate around how you should do document management in an EMR, but many of the EMR softwares store the scanned (or otherwise electronic) documents in the database itself. Over time this can start to make your database very large.

    I know of at least one EMR company which stores the attachments in one database and the rest of the data in another database. This is nice setup since the backup requirements for documents (which can’t be changed) versus data which is constantly changing is different.

    Your point is well made. This is still a lot more fun and easier than boxes of paper charts. However, it’s still an interesting question that I think we’re just now getting ready to deal with.

  • I am an avid opponent of “the cloud”. Everytime there’s a new fad, everybody jumps on it and start throwing around these acronyms and fancy terms. The “cloud” is nothing more than virtualization optimized. In most cases it doesn’t necessarily apply to organizations. What need to be done is a comprehensive analysis of your environment. Take a full inventory of what you have and what you need. Then dissect how to align with your business strategy, driven by compliance protocols. Easier said than done.
    More importantly, avoid bringing in any of the fancy technology fads (“virtualize, send to the cloud, etc”). Those can be quite misleading. When it comes to EMR Data Archiving, optical storage medium is king. Disk is only useful for backups, replication and fast primary clustering performance, tape lacks integrity, speed, and reliability..in addition to migration every few years (among other issues). Optical is still king when it comes to archiving in the healthcare space.

    Phantom Data Systems Inc

  • Why would you be an opponent of the cloud? It has its place and it’s hear to stay. Certainly you can say it’s overhyped and misunderstood, but it can do amazing things.

    Optical is not the best solution for most healthcare facilities. Disk is the best for most ambulatory EMR.

  • John,

    After listening to comments from Health2Con presenters for example, one could more easily see who has a horse in the race or at least a favorite jockey.

    I know you want traffic to your sites, so a little drama doesn’t hurt, but more or even full disclosure from your posters would be helpful.

    Microsoft .Net / Silverlight 3 ambulatory EMR “on the runway”

  • Yes disclosure is an issue sometimes, but most of the time you can see who’s got what interests. I’ve written some posts in the past talking about my connections (which are few). I also do the same when I cover one of my advertisers.

    Yes, traffic is a good thing;-)

  • John,
    If you ready my response closely, my opposition to the cloud lies within the context of tier 3 data archiving. Yes it’s overhyped and misunderstood but it can indeed do amazing things.
    Perhaps you’ve read about the RAID on a data center out west that hosted “cloud services” and inadvertently mixed up various services. This lead to outage of customers not related to the investigation the FBI was conducting. My point John is, measure twice cut once.
    Regarding optical, I stand by my position on the storage platform. However, again you failed to read closely as I was referring to tier 3 storage, not tier 1 or tier 2 (see “Disk is only useful for backups, replication and fast primary clustering performance). As far as archiving goes, you cannot (and I repeat cannot) archive on disk. Extensive research and hundreds of production environments prove this. Ambulatory EMR should be using disk, however when you get into Tier 3 (which is archiving for long term storage), this cannot be accomplished using disk clusters. It’s simply neither cost effective nor compliant in all respects.

  • John,

    If you were to come out, for example, strongly opposed to cloud computing, you would likely be “in trouble” with your cloud platform advertisers.

    I show these blog posts and other EMR blogs to my focus group and they often come away more confused and with less confidence in general.

    Maybe the issue is more along the lines of “advertorials”, or advertising disguised as “white papers” sort of thing.

    Microsoft .Net / Silverlight 3 ambulatory EMR “on the runway”

  • Alani,
    I understood what you were trying to say. However, I’m sure that many of my readers didn’t and so I wanted them to understand that disk is the best option for them.

    The reality these days is that ambulatory EMR software hasn’t reached the point that they’ve needed to archive yet. Disk drives are so large that they haven’t needed to. There’s a real question of whether they’ll ever need to start doing archiving to long term storage. Time will tell.

Click here to post a comment