The following is a guest post by Carl Bergman from EHR Selector.
One thing that’s certain in the EHR world, someone is either polling or blogging about the results. The problem is how do you know which poll to believe and which to trash? It’s not an easy question, if for no other reason than the remarkable volume of studies.
To figure this out, I ask myself five questions about EHR polls. The answers help me figure out which are the real deal and which to ignore. Here my five:
- What does it say? What is its take away? Not just the headline, but what do the findings reveal? A study may be rigorously done, but if the author makes an inductive leap over a cliff from results to meaning, the work is for naught.
- Who’d they ask? A valid poll’s sample should be a microcosm of the whole group. The idea is that if you contacted everyone in the group you’d get the same results you got from your poll.
If the survey lets anyone answer, then it only represents those who answered. For example, let’s say in 2012 Fox News and MSNBC each ran an on line poll of Romney versus Obama. The polls let anyone vote. Would you be surprised that Romney won on Fox, but Obama won on MSNBC?
- What did they ask? If I can read the questions, I look to see if they are fairly worded. I’m leery if they’re a version of the classic leading question, “How long have you been beating your wife?”
- Is it free? I can understand paying for a study that’s cost a lot to produce. What I can’t understand is a study that touts its findings, but puts its methodology behind a pay wall.
- Who did it? If you have questions, you should be able to contact the chief investigator.
Two EHR Poll Examples
Here are two recent studies that make important statements about the EHR field. Let’s see how they fare:
1. Accenture Survey Reveals Most US Doctors Believe Patients Should Help Update Their Electronic Health Records, But Shouldn’t Have Access to Their Full Record. URL: http://goo.gl/2ymctw.
a. The Claim. This poll makes a strong statement about how US doctors view patient’s role in their medical record. It says an overwhelming number of physicians, 82 percent, want their patients to update their EHRs, but only 31 percent believe that patients should be able to see their full record. If true, this has major policy implications.
b. Who Was Asked? Accenture hired Harris Interactive to administer the poll. Harris asked 3,700 physicians in eight countries. This included 500 US doctors. The poll was done on line. Any physician could participate.
The poll’s biggest problem is that it is a self selecting sample. There is no attempt to show that it is representative of US doctors as a whole, much less ambulatory, in patient, etc.
c. Questions? The questions asked aren’t listed.
d. Free? There is no charge for the viewing the poll. The results are posted in two .pdf pages on Accenuture’s site.
e. Investigator. No contact’s given for Harris Interactive. It lists three major Accenture officials.
2. Software Advice: Four Years Later: The Impact of the HITECH Act on EHR Implementations. URL: http://goo.gl/OcIeVO.
a. The Claim. Software Advice is an online technology service for those shopping for vertical software products. Their survey has these major findings:
i. Replacements. 31.2 percent of EHR shoppers were looking for a replacement. It was 21.0 percent in 2010.
ii. New. 16.4 percent of shoppers in 2013 were opening a new practice versus 12.2 in 2010.
iii. Paper. 50.9 percent were dropping paper systems compared to 64.9 percent in 2010.
b. Who Was Asked? Software Advice (SA) polled 385 practices chosen at random from those who had contacted the firm. They were chosen from a group of likely buyers who had contacted the firm. SA is clear about who was in their full group and who they sampled. They say:
i. Self-Selection Bias. Almost all of the individuals we qualified discovered our site through an Internet search and then consented to a 15-minute phone call discussing their EHR selection process. This may skew the results toward buyers who are more technologically savvy, as well as to those who are uncertain as to which product they are going to buy. Buyers who rely exclusively on referrals from colleagues to make their EHR purchase decisions, for example, were not likely to have been sampled. . . . [It also states:]Not included in this survey sample are the countless successful EHR implementations: buyers who purchased an EHR and absolutely love it; or practices for whom the savings in time and efficiency were well worth the costs of the software and the transition.
c. Questions? The questions are not available.
d. Free. Yes. The results are posted on its web site.
e. Investigator? There are no contacts for the survey, however SA’S Larik Malish answered comments from readers.
Of these two examples, Accenture’s claims are based on a self selecting survey, which is unlikely to represent more than those who answered. I wouldn’t give its claims much weight.
SA’s study is representative within its defined limits. Within those limits, it’s worth taking into account.
Trying to make sense of EHR poll claims is not for the meek. There are polls and then there are polls. A few questions can help sort them out.