Common EMR Implementation Issue – EHR Performance Issues

We’re back again with our ongoing series on Common EMR Implementation Issues. Seems like readers really liked my first entry in the series about Unexpected EHR Expenses. To be quite honest, I was really happy with how that post turned out myself. It’s one of the most comprehensive and useful posts I’ve written in the 5.5+ years I’ve been writing about EMR and EHR. Hopefully we can continue that trend.

Today’s Common EMR Implementation Problem: EHR Performance Issues

I have to admit that this is a really tough problem to crack. However, it’s also incredibly common. The symptoms for this problem usually are described as, “THIS EHR IS SOOOOOO SLOW!” (This is appropriate use of ALL CAPS since they are often yelling this.) Followed by a *huff* and an angry doctor or nurse leaving their computer in a fit of rage. Other symptoms might include drumming fingers on the desk while staring blankly at the screen, lots of mouse clicks that get progressively longer and more emphatic, or the sitting back in your chair staring at the screen hoping that something will happen.

Once you’ve identified that there’s a problem with EHR slowness, then begins the fun and exciting (that was written in the sarcasm font) journey to identify the real issue. The biggest challenge with identifying the slowness is that there are a multitude of places that could be the bottleneck that’s causing your slowness. Some of which you can fix, and others you have to rely on your EHR vendor to fix.

To assist you in the ugly process of improving EHR performance issues, here’s a list of possible reasons you could have a slow EHR.

EHR Slowness You’re Responsible For
Slow Computers and/or Laptops – I’ve heard of a few EHR vendors offering free iPad’s with their EHR, but for the most part, you’re responsible for buying the computers and laptops for your EHR implementation. See my “EHR Speed Suggestion – Don’t Skimp on Hardware” below for more info on buying the right hardware. Needless to say, I’ve seen many slow computers be replaced and the EHR went a lot faster.

Slow Local Internet – Your local internet (or LAN as it’s often referred) could be the cause of your EHR slowness. I could have split this point into a half dozen possible issues. Some of them might include: Bad network card, bad cabling, bad switch, bad router, bad routing configuration, bad DNS configuration, overwhelmed network, etc etc.

Of course, in most cases you’ll probably have to call your IT service provider to solve these issues. They should be able to easily test most of the above issues and prove that it works for other internet applications and so it must be some other issue causing your EHR slowness.

Slow ISP (external internet connection) – If you’re using an in house EHR server, you won’t have to worry about this as much (except for interfaces, or EHR updates). If you’re using a SaaS EHR, then this could be a major bottleneck. Good thing is that it’s easy to test your ISP speed. If you’re speed is great to other sites, but not your EHR then you can move on to another issue. If you’re speed is bad for all sites on the internet, you need to see if your ISP can make some changes to provide the speed you’ve purchases from them. Otherwise, you might just need a bigger ISP connection than you have and you’ll be able to get your EHR running much faster.

Also, be sure you don’t have employees using up all your bandwidth downloading illegal (or legal) music or videos. That can eat up your bandwidth really quickly. There’s a reason Netflix uses up 20% of bandwidth on the internet. Movie downloads/watching might be using up your internet connection as well.

Memory on Server – I see this issue most often when a clinic tries to re-provision an old server for their new EHR or when they don’t follow the suggested specs of their EHR vendor. It can also happen when you start your EHR with 1 doctor and then grow your practice to 5 doctors. More users usually requires more memory on the server. There are good tools on servers for analyzing how much memory is being used so you’ll know if this is the problem or not.

Hard Disk Space on Server – This definitely shouldn’t happen in a fresh EHR install, but often can happen over time. Servers don’t like to run out of hard disk space and can do all sorts of crazy and unexpected things if they do. Other things that cause a hard disk to run out space might be backups or large log files. I’ve also seen where the IT administrator takes a 500 GB hard drive and divides it into multiple partitions. One partition for the O/S and one partition for the data. Often they misjudge how much to give to one partition versus the other. So, the one partition runs out of space while the other one has TONS of space left.

Good planning and regular maintenance will avoid these issues.

CPU on Server – I believe this is pretty rare these days since memory is usually the bottleneck instead of CPU. However, if the EHR software isn’t written correctly, this could be an issue. Particularly on older boxes.

Complex Workstation Setup – Your IT service provider might have told you all the great benefits of a thin client setup or some sort of virtualized desktop software solution. When done right, these solutions can work fantastic and save you a LOT of money. When done wrong, they can cause you all sorts of slowness and heartache.

EHR Slowness Your EHR Vendor Must Fix
Slow Server Configuration – There are lots of ways to tweak a server to go faster with less resources. Unfortunately, most of these tweaks are likely going to have to come from your EHR vendor. In a larger hospital implementation, you might be able to work with your EHR vendor to implement some of these tweaks. In a small clinic, you’re basically at the mercy of your EHR vendor to configure the server to run fast.

Slow Server (SaaS EHR) – Yes, SaaS EHR vendor servers can go slow too. The good thing is that your EHR vendor likely has monitoring tools that are watching for any slowness so they can proactively fix it. The problem is that then you’re at their mercy to fix the slowness. Needless to say, an EHR vendor’s server support staff rarely feel the end user pain of EHR slowness. At least the pain isn’t nearly as poignant.

Of course, a chorus of calls from EHR users to the EHR support line will help them understand better and fix the slowness. One call about your in house server doesn’t resonate quite as loud.

Slow or Overwhelmed Data Center Connection – Data Center internet connections are generally quite robust and built with a lot of redundancy. However, since data centers usually host many many different systems, they can also get overwhelmed. Sometimes through spikes of traffic, but more often through other nefarious attacks on the systems in the data center. Often, it’s not even your EHR software that’s causing the issue, but it might suffer the consequence. Not very common, but possible.

A little more common could be an EHR vendor that’s growing so rapidly that they can’t keep up with the demand for their EHR software. Other times the EHR vendor just did a poor job planning to expand their EHR data center services.

Poor EHR Code – Not all code is created equal. Some programmers are good at creating code that will execute quickly, but most are not. Fixing speed issues aren’t trivial. Particularly if you have a large code base that’s been created over a long period of time.

Poor EHR Design – The design of an EHR software often determines how fast it work. Designing for speed from the beginning is crucial. Otherwise, a poorly structured EHR can almost never be made fast.

Related to this is EHR software built on old technology. To use a car analogy, you can only make a pinto go so fast without gutting the engine. Too many EHR vendors are built on engines that can only go so fast. They can keep squeezing a bit more speed out of the engine, but eventually you have no other speed benefits because of the legacy technology limitations.

I’m sure there are other possible bottlenecks. Let me know of any I missed in the comments and I’ll add them to the list.

EHR Performance Finger Pointing
Another big problem with the complex list above is that it often leads to a bunch of finger pointing. Yes, sometimes it will feel like you’re back in Kindergarten again. Your EHR vendor will point the finger at your IT setup. Your IT service provider will point the finger at the EHR vendor. Then, the EHR vendor will point the finger at the hardware vendor. You’ll never be able to talk to a person at the hardware vendor and so you’ll have to use other tricks to prove it’s not them.

Needless to say the finger pointing can get really tiring really quick. Not to mention it can be very expensive as you spend money proving to your EHR vendor that it really is their problem and not your setup.

I’ll follow up this post with another on how to avoid EHR Performance Issues during the EHR selection process. I’ll link to that post once it’s up.

Side Note: This post was much longer than expected. I guess I did have a lot to say about this issue.

About the author

John Lynn

John Lynn

John Lynn is the Founder of the HealthcareScene.com, a network of leading Healthcare IT resources. The flagship blog, Healthcare IT Today, contains over 13,000 articles with over half of the articles written by John. These EMR and Healthcare IT related articles have been viewed over 20 million times.

John manages Healthcare IT Central, the leading career Health IT job board. He also organizes the first of its kind conference and community focused on healthcare marketing, Healthcare and IT Marketing Conference, and a healthcare IT conference, EXPO.health, focused on practical healthcare IT innovation. John is an advisor to multiple healthcare IT companies. John is highly involved in social media, and in addition to his blogs can be found on Twitter: @techguy.

18 Comments

  • Brilliant post John!

    Adding to that list of “EHR Slowness You’re Responsible For” : Installing the EMR/EHR on a server machine which is accessible by staff who think its the perfect place to download songs, watch movies and play games …We once found the culprit of a recurring performance issue to be uTorrent running with MAX DOWNLOAD SPEED and MAX CONNECTIONS set to 0 (Infinite :)) … Every evening at 6:30 the EMR would start performing slowly, as someone would use the server machine where it was installed post 6:30 to download “Awesomeware”

  • @Nrip – yes, it is important to have the proper policies in place to not allow downloading of music…to any PC in the office, really.

    A staff member loading uTorrent on any pc, especially the server is a major issue, not only on the performance side, but on the security side.

    On weak equipment – too many offices fresh off the pain of buying a new EHR skimp on the hardware to then run it.

    I’m of the philosophy of not buying top of the line nor bottom. Buy in the middle and you’ll get many years of use.

    Poorly written code is a constant issue from the “big boys” to the smaller scrappy players. The problem is you won’t know about this until it is too late. Your only saving grace is to talk to a practice, of similar size, to see what, if any issues they are having.

    Finally: if your practice has more that 5 users, a web based system is probably not for you.

    Everyone is free to defend their system against that statement…I base this on real life experience.

  • Thanks for a more technically minded post John, I always look forward to those. I think there is one slowness issue that you didn’t identify though, the user and their familiarity with computer systems. One bitter pill that I force most folks to swallow that are starting an EHR for the first time is that THE EHR WILL BE SLOWER THAN PAPER. Period. The longer your practice has been running on paper, the slower it will be to adapt to a computerized system. I’m not just thinking about the people who hunt and peck on a keyboard (as most EHR’s ought to be click-friendly by now) but the simplest concept that the little mouse cursor on the screen represents you, the user.

    As you’ve said though, the benefits of an EHR system over a paper system are worlds apart despite these issues. I hope you’ll explore this dilemma a little more.

  • Andrea Morgan,
    You’re welcome to borrow the sarcasm font. As long as you don’t apply it to the nice complement you put in your comment above.

    Nrip,
    I talk about the uTorrent like downloads in the IPS section. I expect more of those happen on desktops than servers, but certainly either is possible.

    John,
    I’d say some of the newer web based systems could do well with more than 5 users. Just depends on the EMR and the needs of the practice.

    Andrew,
    I like to bring back my inner nerd sometimes. I’ve had to change my profiles to now say Former Nerd. I’ll see about more technical ones in the future.

    I wouldn’t say that an EHR is always slower than paper. Do you remember my post a while back about the medical student friend of mine that hated charting on paper since he can type faster than he can write? I get your point, but as the new generation of typers hits the medical field I think that will affect this view a bit.

  • One of the best things we did this year was to dump our server and thin client terminals and plug standalone desktops directly into the ethernet. Completely sped up and transformed our SaaS-based EMR performance. So nice now!

  • Dr. West,
    Actually I think your experience might have been part of why I thought about the complex desktop setup. I can’t say I thought of it specifically, but I knew I’d heard of some doctors having trouble with it.

  • In a server based install, I’m actually a fan of thin clients.
    No residual PHI to worry about.

    Thin clients on a SaaS EHR is not desirable, as has been mentioned.

    The challenge lately with thin clients is the graphical requirements are getting such that a thin client either doesn’t meet the min requirements or the cost difference between a thin vs. fat client are very slim.

    Thin clients have many security advantages, yet have operational dis-advantages.

    New SaaS EHRs might do OK for a practice, but I have 2 clients on different SaaS EHRs. Both offices have about 8 people in the office.

    One office absolutely hates the slowness of their EHR, the other is OK with the speed.

  • We put a lock-down policy into effect before we even went down the EHR path. If it is not pertinent to business, they cannot access it – PERIOD!

    Personal e-mail is allowed – via WEB ACCESS ONLY – during BREAKS and LUNCH – at no other time.

    NO SOCIAL NETWORKING. NO NON-MEDICAL VIDEO. NO MUSIC. NO EXCEPTIONS.

    If the firewall, content filter or other scans block the access to the attempted site, no problem – look at it when you get home. No, the cute panda baby in the zoo video feed is not an exception. No, the mother bear with her cub cam is not an exception. Don’t ask again!

    The biggest complainers, aside from the “Can’t I PLEASE listen to so-and-so in the morning, no one else has to know . . .” are the big bosses – the CEO, the CFO, the Head of Nursing, etc. They insist on being able to access social networking sites, being able to purchase theatre tickets, being able to watch TV on their computers, and listening to music downloads.

    I locked down, via an external referral DNS source, which, when inappropriate content is accessed, brings up a company logo, politely states that the content the person is attempting to access is not allowed based on REASON INSERTED and CONTENT TYPE, and gives them a clickable e-mail link to send a message to me if they disagree.

    After six months of being yelled and screamed at, refusing to make the requested changes – including video and music feeds [NO! Go buy a CD player for your desk.] and being threatened with my job, I prevailed – but only after reminding management that Federal law was on my side and sounding like a broken record by repeating several thousand repetitions of “just because you are the President or VP of Finance doesn’t give you the right to circumvent the written policy, you are subject to the same policies we put in place for “the peons’,” did they quit pestering for “special privileges.”

    As a CIO, my job is to make the networks perform at peak. My mission is to protect the data stored on those networks at all costs and NO ONE has the right to force me to give them any special privileges – no matter who they are or what position they hold.

  • I’ve worked with many vendors as well as many many providers. If you draw up a matrix, you will invariably find that one vendor will have happy clients where speed and slowness is not an issue and the same client will have clients that complain of the same issues.

    Why is this?

    Is it the vendor? Is it the Practice? Perhaps both.

    The problem is, in most cases, it is bad customer support, bad customer relationship management.

    As all providers know, you need to identify and isolate the problem, proper diagnosis, order tests, and then try solving the problem – proper drugs, proper treatment plan.

    Vendors need to learn from their clients.

  • Nrip Nihalani: Don’t see a way to contact you personally, but maybe the mods here can forward you my e-mail address.

    Please feel free to use anything I posted in your blog. The sooner we can teach the masses who work in Healthcare that security must be taken SERIOUSLY, the better off both those trying to protect the networks, and EHR in general, will all be.

  • Battered CIO,
    Thanks for sharing your war stories. I still love how we think many of the things you listed are ok at work. Far too many people don’t realize that during work you’re suppose to work instead of dealing with other non-work stuff.

    Of course, with cell phones basically being mini-computers, this is a challenge that will only get worse.

    I could definitely forward you his contact info, but it seems like Nrip already left it in the comments.

  • Electronic Health Records are all critically dependent on network and application performance. As John pointed out, there is a long list of impairments that cause slow EHR delivery and cranky physicians. Any drop in bandwidth or increase in latency can make these applications unusable or inaccessible, potentially compromising patient care. The responsibility of ensuring EHRs performance from site to site can be painful when you lack visibility into the network connecting centralized resources to remote offices – including WANs, service provider cloud networks and remote LANs.

    AppNeta has been working with John Halamka and his team at Beth Israel Deaconess Hospital . Halamka is also author of the blog Life as Healthcare CIO, where he has pointed out that it is critical to proactively pinpoint, not only where the network is slow, but why the it is slow.

    To successfully assure the performance of EHRs, and continuously monitor them to mitigate slow, poor performing service, you must have the ability to:

    Pre-assess the network to ensure the migration to electronic health records goes smoothly
    Quickly identify performance problems within complex deployments
    Maintain and optimize performance of wireless PCs and other devices
    Protect yourself and your staff against unpredictable connectivity issues and loss

    PathView Cloud’s zero administration micro appliance enables remotely managed end-to-end visibility to monitor the performance of EHRs from the perspective that matters most, your finger pointing end users. To check out more about PathView Cloud at http://www.appneta.com, or sign up for a free trial.

Click here to post a comment
   

Categories