TURF: An EHR Usability Assessment Tool

The following is a guest post by Carl Bergman from EHR Selector.

To paraphrase Mark Twain, everyone talks about EHR usability, but no one does anything about it, at least until now. Led by Dr. Jiajie Zhang, the University of Texas Health Science Center at Houston’s National Center for Cognitive Informatics and Decision Making (NCCD) has developed several tools for measuring usability.

Now, Zhang’s team at NCCD has put several EHR usability tools into a Windows based app, TURF, an acronym for Task, User, Representation, Function. Funding for the project comes from ONC’s Strategic Health IT Advanced Research initiative.

TURF’s Tools. TURF has two major tools, Heuristic Evaluation and User Testing:

  • Tool One. Heuristic Evaluation: Expert Screen Capture and Markup. This tool takes EHR screen snapshots and let you compare them to usability standards. You can markup the screen and document the problem.
    Turf Expert Markup Tool - Showing Problem and Documentation
    For example, you can note if the error is minor, moderate, major or catastrophic. The system has a review function, so others can look at your markup and comment. The system also compiles your edits and can generate various statistics.

    • Administration. To work with groups, the system has several preset admin template forms and a template editor. The furnished templates cover these areas:
      • Demographics
      • Expert Review
      • Performance Evaluation, and
      • System Usability. This form asks 10 questions about the EHR, such as:
        • I think I would like to use the system frequently,
        • I thought there was too much inconsistency in this system,
    • Standards. The system uses the National Institute of Standards and Technology’s (NIST) EHR usability protocol, NISTIR 7804. You may also add your own rules to the system. (Also, see EMRandEHR.com, June 14, 2012.)
    • EHR Sections. Using the NIST protocol, the system’s review areas are:
      • Clinical Decision
      • Clinical Information Reconciliation
      • Drug-drug, drug-allergy interactions
      • Electronic Medical Administration
      • ePrescribing
      • Med – Allergies
      • Medications list
      • Order Entry
      • User defined
  • Tool Two. Live Session Testing. TURF’s user test tool sits on top of an EHR and recording each movement. TURF’s designers have created a system that not only tracks use, but also adds these major functions:
    • User Sessions. TURF captures live screens, keystrokes, mouse clicks and can record a user’s verbal comments in an audio file.
    • Administration. The tool is designed for testing by groups of users as well as individuals. It captures user demographics, consent forms, non disclosures, etc. All of these can be tailored.
    •  Testing for Specifics. TURF allows managers to test for specific problems. For example, you can see how users eprescribe, or create continuity of care documents.
    • Comparing Steps. Managers can set up an optimum selection path or define the steps for a task and then compare these with user actions.
    • Reporting. TURF builds in several counting and statistical analysis tools such as one way ANOVA.

  • Running TURF. TURF isn’t your basic run and gun app. I downloaded it and then tried to duff my way through, as I would do with most new programs. It was a no go. Before you can use it, you need to spend some time setting it up. This applies to both its tools.

    Fortunately, TURF has about 30 YouTube tutorials. Each covers a single topic such as Setup for Electronic Data Capture and runs a minute or so. Here’s what they cover:
    Turf Tutorials Screen
  • Hands On. Installing TURF was straightforward with one exception. If you don’t have Microsoft’s .Net Framework 4.5 installed, put it up before you install TURF. Otherwise, the install stops for your to do it. TURF will also want the Codex that it uses for recordings installed, but the install deals with that.

    TURF is a Windows program, so I ran it in a virtual Win 7 session on my iMac. Given the environment, I kept the test simple. I ran TURF on top of a web based EHR and had it track my adding an antibiotic to a patient’s meds. TURF stayed out of the way, recording in the background.

    Here’s how TURF captured my session:
    Turf Playback Screen
    The left side screen played back my actions click for click. It let me run the screen at various speeds or stop it to add notes. The right screen lists each move’s attributes. You can mark any notable actions and document them for review by others. You can save your sessions for comparisons.

I found TURF to be a versatile, robust tool for EHR usability analysis. Its seeming complexity masks an ability to work in various settings and tackle hosts of problems.

If you aren’t happy with your EHR’s interface, TURF gives a remarkable tool to show what’s wrong and what you want. Indeed, with some adaptation you could use TURF to analyze almost any program’s usability. Not bad for a freebie.

About the author

John Lynn

John Lynn

John Lynn is the Founder of HealthcareScene.com, a network of leading Healthcare IT resources. The flagship blog, Healthcare IT Today, contains over 13,000 articles with over half of the articles written by John. These EMR and Healthcare IT related articles have been viewed over 20 million times.

John manages Healthcare IT Central, the leading career Health IT job board. He also organizes the first of its kind conference and community focused on healthcare marketing, Healthcare and IT Marketing Conference, and a healthcare IT conference, EXPO.health, focused on practical healthcare IT innovation. John is an advisor to multiple healthcare IT companies. John is highly involved in social media, and in addition to his blogs can be found on Twitter: @techguy.


  • Hmm.
    How’s this for a test: how many clicks does it take to get 1 meaningful use counter?

    I really don’t think a test like this is needed.

    The real problem is EHRs, for the most part, have not been demand driven.

    Docs (mostly) have only flocked to EHRs to get a government check…not because an EHR increases efficiency.

    Notice how many dentists are voluntarily on EHRs?

    If EHRs where “all that” there would never have been a need to bribe docs to go electronic…and we wouldn’t have this monster called Meaningful Use.

    The usability of an EHR is easy to point out…fixing it after the fact is another story.

Click here to post a comment