This week I’ve been inundated with press releases and companies promoting their Best in KLAS ranking/rating (whatever you want to call the award). In fact, I’ve gotten so many notices from so many healthcare IT companies, it prompted to me to send the following tweet:
Given all the press releases I've gotten, are there any health IT companies that aren't Best in KLAS? #HITsm #hcsm
— EMR, EHR and HIT (@ehrandhit) January 29, 2015
Ironically, that tweet was before I got another dozen more press releases, blog posts and tweets informing me of how great their company is because they’re “Best in KLAS.”
In a conversation I had with someone who was “Best in KLAS” and when I saw a blog post by a CEO that was so proud of their Best in KLAS rating, I wondered why we don’t have some expectation of financial disclosure with these type of ratings and awards. This isn’t an issue for KLAS alone, but would apply to Gartner and a number of other organizations that offer these type of healthcare IT software ratings.
In the blogging and media world, the topic of disclosing financial interests is often discussed. It’s a policy that I follow myself. If I ever write about a company for which I have a financial relationship (advertiser, sponsored content, email campaigns, etc), I disclose that financial relationship in the article. I believe it’s important for anyone reading that article to know that there’s a financial relationship which could sway the content.
Shouldn’t we expect the same from companies who have a financial relationship with these ratings organizations? There’s a possibility that the financial relationship could have made a difference in those ratings. Shouldn’t we know about this potential for bias?
Of course, I don’t expect we’ll see many organizations take me up on this idea that they disclose their financial relationship. So instead, I’m calling on those Best in KLAS companies who don’t have a financial relationship with KLAS to come forward and disclose that they don’t have a financial relationship with KLAS and they still were given a Best in KLAS rating. I’ll be interested how many come forward.
As I’ve long told people who ask me about the value of KLAS, I think there’s so many ways to skew their results that I don’t put much value in their results. Plus, I’m not sure about their methodologies which include doing ratings at EHR user conferences (biased sample anyone?).
However, for marketers, I tell them they absolutely should make the most from a Best in KLAS rating. Most healthcare organizations don’t understand (likely because they’re too busy) the nuance in proper ratings and therefore blindly use KLAS for their decision making. Unfortunately in healthcare IT, these people don’t have any other choice but KLAS. So, given no better alternative, it’s no surprise that they use what’s available.
Of course, my hope is that most healthcare organizations use KLAS, Gartner magic quadrants and whatever other ratings and rankings that exist as just another data point. Triangulating across those and your colleagues is often going to lead people to the best solution.
Full Disclosure: I have an affiliate partnership with a company Gartner bought for some EHR lead generation. I’m sure they’ll love this article.
We at Civerex long ago decided to not respond to RFPs – we had the suspicion that many of the issuers had made up their minds before their RFPs came out.
We find that organizations that buy on “features” many times have no idea with they will do with a product (healthcare, manufacturing)and no idea what to look for.
They end up buying the product with the most checkmarks and never seem to get around to using the product to increase throughput and productivity.
I don’t know if vendors “buy” ratings but I agree with you that many of the vendors say ‘ours is best’.
We “announced” a non-existing feature one time as a test and counted five vendors who introduced the same “feature” two weeks later.
I worked for GE and, along with everyone else at GE, read about the mid 1960s product insert on “turboencabulation”, that actually received calls.
One time I went on site to solve a problem and met with the CEO to ask him to describe “the problem” in his words – the response was “never mind the problem, just give us the solution”.
Mr. Spock would say that was not logical.
Karl,
Great story about a fake feature. Good stuff. The analysis of RFPs is really interesting. Although, in the hospital setting do they buy anything that doesn’t go through an RFP?
John, if you get access to the KLAS ratings, you will see that our little EHR/PM company has higher scores – in every category, often by a lot, last time I checked – than anyone in our section. In fact, our scores are generally higher than any vendor you will find scored by KLAS anywhere. We have had the most dominant scores from KLAS for years.
We received our scores well before we paid KLAS any money. And, even though we have paid them $$ for years (in order to access the customer-level response data), we have never received a “Best in KLAS” score. From what they tell us, over and over again, our specialty specific nature compels them not to compare us to the behemoths who try to appeal to everyone.
At present, we no longer pay KLAS. Their business model has changed and they told me specifically that they want to back away from using the KLAS scores as marketing materials. They want to focus on consulting with companies to improve. There isn’t much that KLAS is going to teach us about customer service…in fact, it’s more likely we could teach them.
I have a lot of problems with KLAS and how they work. We have been very vocal about their problems (I spoiled their twitter-fest last year at this time calling them out) and have even explored legal action when they mischaracterized us from time to time. But I’ve never felt that writing them a check affected our scores one way or the other. In fact, we’re the poster-child for proving that KLAS isn’t pay-to-play.
I’m glad to be transparent and tell people what we paid and what we got for it. Perhaps instead of going only after KLAS, we should be asking the vendors? One could easily find out when vendors started to pay KLAS and whether or not scores changed.
As for taking surveys @ EHR users conferences…my distinct understanding is that they disallowed that option years ago unless a KLAS representative is there to watch the process. Given that they call and interview for the data *and* they require your customer list, I don’t think that’s an issue.
Chip,
I think the article does go after vendors to disclose their financial relationship and not KLAS. Although, I think they both should in the interest of transparency. I don’t think that payment is always foul play, but there’s the potential for foul play, so people should know who is paying and who isn’t paying. I think most of my readers still read my content even when it’s a sponsored post or I talk about an advertiser because they know that it’s my reputation on the line as well. In fact, I think the financial disclosure adds more credibility to whatever you’re sharing.
Thanks for sharing how it’s gone for your organization. Considering the number of vendors affected, I’m sure there are stories that go from all angles.
I have participated with KLAS at two different companies and dropped them after experiencing several issues with them that I believe resulted in highly suspect results:
1) While not necessarily “pay to play”, at one time (and perhaps still today) KLAS would attend (if compensated extra) user meetings for paying vendors to gather surveys. It doesn’t take a rocket scientist to know that users that pay to attend their vendor’s annual meeting will tend to be much more satisfied with their vendor than the average user.
2) In 2012 KLAS published a press release that Practice Fusion was ranked #2 rated Cloud EHR. The rating was driven by the technique above as well as some other poor statistical techniques. If I remember correctly many vendors questioned the methodology and KLAS revised the report – but the press release was out and I don’t believe they issued a retraction to the press release (or if they did not with the same fanfare as the original distribution).
3) KLAS’ methodologies are not statistically valid and actually can be easily manipulated by vendors. For example, at one time (and probably still today) it was well known that certain vendors would do things like only submit a subset of their customer base to KLAS – limiting the customers that KLAS was contacting to those the vendor thought most likely to rate them highly.
I decided the “Best in KLAS” was at best flawed and and worst rigged and there are much better and less costly ways to accurately survey your customers.
At risk of appearing to be a KLAS apologist (nothing could be further from the truth, given our relationship), I offer the following:
– KLAS has never charged us to have one of their staff visit our users’ conference to oversee the survey collection. And although you may find many of a vendor’s best customers at a users’ conference…you may also find their worst. When you stand in front of hundreds of your clients and say, “KLAS wants your input,” you can be sure that anyone with a gripe is first in line.
– As I understand it, the results from the surveys are not automatically included, they are followed up with an interview. The comments from these interviews were always the most valuable thing KLAS did for us, frankly. Our clients would tell them things they won’t tell us.
– KLAS would not report our results until we gave them a _complete_ list of our customers. Not the happiest. Not our filtered version. In fact, KLAS (at least used to) make a decided effort to get the names of clients *from other clients*. We showed up on the KLAS radar as the result of practices telling them about us, not the other way around.
– That Practice Fusion appeared highly in their list just means that they found similar results to almost every other EHR survey I’ve seen. The free/cheap EHRs are often at the top of the list.
Ultimately, I do agree that the Best In KLAS awards are deeply flawed for many reasons. I don’t think they are rigged, though. There’s a big difference. We have been approached by other ratings/review sites that are _explicitly_ pay-to-play (literally, “If you pay $X we can guarantee you a good review…”).
Let me tell a real story. When we first got rated years ago, I tracked down a copy of some of the comments. One of the comments in there was patently false, but in a positive way – it said something like, “PCC only hires staff who have worked in pediatric offices…” It’s a nice thing for potential customers to believe, but not at all true.
So I called KLAS. I got a woman on the phone an explained that there was something wrong with one of the comments. She asked for details and I was in the middle of a sentence when she said, “Wait. You’re the vendor? I can’t talk to you. Period.” I tried to explain I wasn’t calling to complain or or ask them to take something BAD down, I was calling to correct something that made us perhaps look better than we should. “It doesn’t matter. We take no input from the vendors.” And then she said goodbye and hung up.
I always found that interesting.
Now that we no longer pay them, if I learn that our scores go down without good evidence, perhaps I’ll change my tune. But they were high before we engaged them, I expect them to remain high now that we’ve stopped.
I’ve heard from a former KLAS researcher that it isn’t about pay to play with KLAS, but your relationship with them does matter and influences the ratings.
Chip,
How do they know if you’re giving them the complete list of customers or not? I’ve not found what you said about disgruntled customers at user conferences. I’ve found very few disgruntled users at user conferences and the few I’ve found have to talk under their breathe because they know that so many around them don’t agree with their take and so they’re afraid to share their gripes. I’m sure there’s the exception, but that’s been my experience at many user conferences.
I actually think KLAS has the best of intentions and the policy of the lady hanging up on you illustrates some of the extreme measures they try to put in place to be unbiased. However, their business model gets in the way of being unbiased. Maybe that’s why they’re trying to grow the consulting arm of the company like you mentioned.
John:
How do they know they have your list? You affirm as such and it’s easy to figure out. They ask the people they talk to for the name of other practices.. And if they don’t get a large enough sample, they either don’t score you (which is what they did to us) or you get different degrees of confidence. That’s Data Analysis 101, any vendor in this business is going to get called out on it. YES, I’ve had problems with some of the results I’ve seen from them but their scores generally reflect what I see every day more than anyone else. We get a lot of calls from clients of National Vendor A, their KLAS score is down the next quarter.
As for complaints at a users’ conference…I’ve had some _really long and frightening_ classes, believe me. And I’ve been the disgruntled user myself. I just shrug – we had our scores long before we ever handed out a survey at a UC and we’ve never paid KLAS to attend our UC. And we’ve never handed them out at a UC without someone from KLAS there. That’s all I know.
I fully agree that they’ve painted themselves into a corner. Are they in the marketing data business or not? If they aren’t – which is what they told me 2 weeks ago – then why have a “Best In KLAS?” And if the marketing of the ratings does so much to undermine their own brand…they’ve got a problem. I had talked to them about doing cool things like having an RSS feed of vendor data, etc., or having some truly great specialty-level focus. No interest.
I couldn’t agree more, though, about transparency. If you want, I can look up what we paid them and when and will publish it here.
Here’s one “hole” in their business model that they really don’t like anyone to talk about: all it takes is a single doctor to get all of their data. If you are a physician and raise your hand to give them your own survey information, you get access to their scores. All those PCC clients who filled out their surveys? All I’d have to do is ask. Every other EHR vendor is in the same boat. Just sayin’.
“I couldn’t agree more, though, about transparency. If you want, I can look up what we paid them and when and will publish it here.”
That’s my real message in this post. Let’s just have some transparency. If you’re a vendor that’s paying a million dollars to a company and they’re giving you good ratings that might matter to someone who’s looking at those ratings. It’s the exact same as someone who pays me any money to advertise on this blog. It could certainly sway what I write and so people should know about the payment and they can evaluate if I’m a shill for that company or not.
As for you publishing what you’ve paid, no need to do it here. As I said in the post, I’d just love for you to disclose it in any KLAS releases you do. Interestingly, I think it will raise the credibility of your release if you say you’re not paying them for the survey and you were still rated high.
Ah, and therein lies the rub. We don’t pay KLAS any more. So any scores we show have to be old or surreptitiously acquired.
You make an interesting point about stating that we don’t pay KLAS. I have to think about that.
Thanks!
Good point Chip. So I guess anyone that announces their Best in KLAS paid?
Not necessarily, as the award is announced, so you could easily just reference the BIK award with the date. However, if you want to get the full spread of data (they rate vendors in ~30 categories), then you have to pay.
I don’t think that any of the BIK winners from the sections we’ve been scored in has had a higher score in _any_ of the sub categories than we have. Maybe ever, but certainly not recently. Sometimes, the gap is giant. I could give you examples from the recent scores…but we don’t have access any more 🙂
Is this sector of software possibly one of THE most important and at the same time the most controversial and by some the most “hated”?