Testing: An Undervalued Aspect of a Successful EHR Implementation

The following is a guest article by Matt Hawkins, EVP Sales and Marketing at Boston Software Systems.

A large regional health system implemented a “Top 5” EHR solution for their Gynecologic Oncology department. However, three years after implementation, staff were still using paper charts while the EHR sat virtually untouched. Why? The new system was missing several key components including: specialty specific templates, medications, chemotherapy infusions, and billing codes. How were such critical features missed? A lack of user testing.

Testing is an often overlooked and underappreciated aspect of any systems implementation. Because it is often the final step before a Go-Live, it is often rushed in an effort to bring the project over the finish line. When planned and executed properly, testing can add tremendous value to a project AND can de-risk your technology investments. Had this Gynecologic Oncology department adopted a solid testing plan, they would have uncovered the missing features early, not three years into the project plan.

So what are the keys to a successful testing phase? Here are four.

Clinical Alignment and Usability

Too often, clinical use and alignment to workflow is an afterthought during EHR implementation. The sales team inadvertently makes promises the tech team cannot carry out. This is because integration requires people who understand all of the relevant perspectives, in order to incorporate the right information into the customized model. When clinical tasks are mapped out as a sequence of events to capture workflow and interdependencies with systems, people and process, a foundational understanding of the gaps are uncovered. This allows attention to be shifted to the correct alignment of vendor solutions and the needs of role-based users. In many cases, there is still time to justify ancillary solutions that “fit” the expectation of the technology to the end “result” of usability.

Framing Appropriate Expectations

Out of the 96% of physicians who have adopted EHRs, 66% say the newly installed technology has fallen short of expectations. Consider a different strategy in the testing phase, to prevent ongoing usability issues and lingering gaps in efficiency and productivity.

Involve physicians and role-based users from each department as critical stakeholders in testing for success. This is one of the most important aspects of framing expectations. These super users can attest the system will work as designed and for clinical use, feel more involved in the process and assist in controlling the talking points around the testing instead of letting rumors run wild.

Project Plans

Make sure there is sufficient “slack” in the project plans to allow for changes identified during testing. Before unleashing it in a “live” practice environment, you will want to look at critical success factors for evaluation purposes. There are specifics to consider:

  • Build in effective and timely approaches for efficiency improvement by creating a working roadmap that outlines repetitive, manual-based processes.
  • Critical processes should be unit tested once they are fixed, then integrated testing should be performed to confirm no other components or workflows were impacted by the fix.
  • Performance data requires large data sets. Testing teams may not have access to these data sources. If you’re testing “fake” or manually entered data, it may not be a “real” representation of functionality.

Consider using a 3rd-party vendor to optimize inputs and improve project timelines.  Success depends on timely approaches for managing the complex array of data from multiple information sources. The highest level of operational productivity must be measured. The goal of future usability and sustainability is to have patient data available to anyone who needs it, when they need it, and where they need it.

Kathleen Kostowny, RN, PMP and EHR Implementation Consultant commented:

“We want to ensure we are good stewards for qualitative and quantitative testing. Collaboration with key stakeholders will promote future usability and help to ensure a solid/reliable process going forward.”


Another key to testing is spending time ensuring your data extraction is tested and usable as well. Many EMR vendors provide data extraction tools for importing data from legacy systems to a new EHR. These services fall in the, “yes we do that,” category. It’s best to define what exactly this statement means, the cost, and how it impacts role-based users. What seems like a simple fix of manually entering demographics can turn into months of manual data entry, and an overwhelming data burden. Vendors use outdated technology and substandard practices to check the box of moving data, but they don’t move it well. Oftentimes it’s not accurate, and requires manual interventions that aren’t even realized until Go-Live. It’s like a moving truck pulled up in your driveway and dumped out everything in the street, with no help or organization. Don’t add the complexity of manual data entry plus additional testing, if there is a better option.

Shahid Shah, Founder of Netspective Media shared:

“The success of any software brought into a medical environment depends on these factors: (a) the quality of the vendor creating the software, (b) the static part of the software you buy, e.g. what doesn’t change across the vendors’ customers, (c) the dynamic and customizable parts of the software you buy, e.g. what changes in the software once you install it in your environment, (d) how you educate your users and train them to use the software. Through RFP responses and demos you can easily evaluate factors (a) and (b) before you purchase the software – and you can count on the vendor to be very helpful. However, factors (c) and (d) are directly your responsibility – you must create a separate environment to install the software, make your customizations and adjustments, then train your staff on those adjustments and customizations. The combination of creating your own environment, doing the customizations and training is what’s referred to as “implementation testing”. The vendor can guide you with implementation testing but they cannot do the job for you. Lack of implementation testing is the single biggest reason for the failure of any software platform within an enterprise”. 

To circle back to the Gynecologic Oncology department mentioned at the beginning of this article, quite a lot of additional customization was necessary, to create a system that worked for role-based users. We created a roadmap, based on user feedback. This included data migration, shortcuts, and hardware integrations to make the system more intuitive and connect the gaps. Medications and pharmacy data were updated, chemotherapies were included, and appropriate billing codes were aligned, enabling physicians to treat patients with less disruption to clinical workflow. Testing was improved then repeated with the physicians and clinicians. Their involvement in the process and the re-testing, ensured success, enabling 150 users to quickly grow to 250 users, and not only were the physicians able to see more patients each day, their improved documentation became more valuable for clinical research.

Before committing to Go-Live, have a strong testing plan that includes a thorough understanding of the current state of your organization with an identification of the key goals for improvement.

Working with hundreds of clients at Boston Software Systems, we’ve seen the strategic benefit to ensuring that a data migration runs like clockwork. While we often have clients come to us when their testing and data migration is “off the rails,” it’s much easier to start with appropriate testing and planning from the start. Make sure you have a platform that turns testing around so that Go-Live and future use of the EHR can run smoothly. EHR testing is never a “one and done” endeavor.  If you don’t have expertise in-house, consider what experienced partners can help ensure project success and Go-Live readiness.

About Matt Hawkins

Matt Hawkins, EVP Sales and Marketing at Boston Software Systems, has worked with hundreds of clients to keep their data migrations from heading off the rails. Matt is passionate about bridging the gaps left by other software vendors. Visit our hashtag #bridgethegaps on Twitter and stay in touch with us @Bossoft to learn more about what makes us the leader in healthcare automation solutions for the last 30 years.

Note: Boston Software Systems is a proud sponsor of Healthcare Scene.

About the author

Guest Author

Guest Author


  • Absolutely! I have been tooting my testing horn at every site I contract or hire on as an IT employee. It’s the fail safe switch and the more testing we do upfront, involve clinical users in the testing event and pad the project timeline to allow for retesting of issues the better our implementation outcomes will be.
    Well done! Thank you!

Click here to post a comment