Lessons Learned from our EMR Upgrade – Part 2

As I discussed in the last blog post we were very busy this past spring with our biggest EMR upgrade to date:

  1. Upgrade from the 2005 software to the 2010 version (2011 upgrade delayed on the advice of our VAR), making a big jump.
  2. Purchase new servers and new memory (SAN)
  3. Switch to virtual servers / VMware
  4. Convert our database from 2-database structure to single database to accommodate the 2010 software.

This was more of a system replacement than an upgrade.  The only parts that weren’t completely replaced were the network components and some peripheral applications (web portal and document scanning).

Despite realistic expectations the upgrade took longer than expected.  Some problems took many weeks to solve.

  1. Despite a successful test run, the dual-to-single database conversion was fraught with problems and took longer than expected.  The computer that was running the conversion software (called a “migration tool”) had a RAM failure during the operation, which slowed the conversion down but didn’t kill it.  When we saw the operation slow down we had a dilemma – do you stop to troubleshoot or let it keep running slowly?  We have over 250,000 patient records in our database so the conversion was expected to take well over 72 hours – longer than a weekend.  That meant we were already looking at EMR down time during office hours.  We stopped the migration to diagnose and replace the RAM.  Then the migration tool itself failed, forcing another interruption and requiring our vendor to troubleshoot and patch the migration tool.  The migration tool is an unusual piece of software.  You only need it once so about the time you have learned to use it you don’t need it anymore.  On the vendor side, every customer’s database / hardware situation is different, so the migration tool is never totally debugged.  That is why we delayed our upgrade so long – we wanted the vendor to gain some experience with the migration tool before we used it.  We were still by far the largest database conversion they had ever done.  In spite of the difficulties the result was an intact single database that gave us no further trouble once the migration was completed.
  2. Another contribution to our delay in upgrading was waiting for our vendor to support VMware and give us hardware specs.  Even with that accomplished VMware was a nightmare to set up.  Performance was very slow initially and took days to correct.  The biggest problem was the printers.  Printer preferences were lost several times a day and it was not unusual for my documents to get printed at a member practice across town despite having reset my printer preferences several times that day.  That wreaked havoc on clinic operations and took over a month to fix.
  3. We were blindsided by a bizarre “failure” of a T1 line to one of our offices.  The line was somehow put in some sort of diagnostic mode, rendering it unable to function but showing it as normal to our monitoring.  For days we assumed that office’s performance problems were related to the upgrade.
  4. Some issues were purely our fault.  We did not adequately staff our upgrade operations.  We had only our chief operating officer and our IT specialist to handle problems and questions; they couldn’t get off the phone long enough to fix anything.  This also impaired communications significantly.  To make things worse each of them had immediate family members become suddenly ill, requiring that they take some time off during the upgrade.

The next post will be my analysis of this great adventure.

 

About the author

Dr. Michael Koriwchak

Dr. Michael Koriwchak

Dr. Michael J. Koriwchak received his medical degree from Duke University School of Medicine in 1988. He completed both his Internship in General Surgery and Residency in Otolaryngology-Head and Neck Surgery at Vanderbilt University Medical Center. Dr. Koriwchak continued at Vanderbilt for a fellowship in Laryngology and Care of the Professional Voice. He is board certified by the American Board of Otolaryngology-Head and Neck Surgery.
After training Dr. Koriwchak moved to Atlanta in 1995 to become one of the original physicians in Ear, Nose and Throat of Georgia. He has built a thriving practice in Laryngology, Care of the Professional Voice, Thyroid/Parathyroid Surgery, Endoscopic Sinus Surgery and General Otolaryngology. A singer himself, many of his patients are people who depend on their voice for their careers, including some well-known entertainers. Dr. Koriwchak has also performed thousands of thyroid, parathyroid and head and neck cancer operations.
Dr. Koriwchak has been working with information technology since 1977. While an undergraduate at Bucknell University he taught a computer-programming course. In medical school he wrote his own software for his laboratory research. In the 1990’s he adapted generic forms software to create one the first electronic prescription applications. Soon afterward he wrote his own chart note templates using visual BASIC script. In 2003 he became the physician champion for ENT of Georgia’s EMR implementation project. This included not only design and implementation strategy but also writing code. In 2008 the EMR implementation earned the e-Technology award from the Medical Association of Georgia.
With 7 years EMR experience, 18 years in private medical practice and over 35 years of IT experience, Dr. Koriwchak seeks opportunities to merge the information technology and medical communities, bringing information technology to health care.

2 Comments

  • I am interested in working with you.
    I have a Helath Informatics Certification.

  • Good day sir,

    I’ve also had some experience with migrating data in our hospital. I’ve noticed that you purchased new servers along with a computer that does the migration. I would suggest that a good strategy for migration is to have the two systems (old and new) run in parallel (the old system in the old server and the new system in the new one) and run a script in the migration computer that synchronizes data to the new server. Once the 2 have the same data, you take the old one out and your employees won’t experience any down time.

Click here to post a comment
   

Categories