On June 27th the Petya Malware (or NotPetya or ExPteya) struck Nuance Communications (NASDAQ: NUAN). For days the company’s eScription speech-recognition platform were unavailable, forcing thousands of healthcare clients to find alternatives for their medical transcription. During the crisis and in the weeks that followed, Nuance borrowed a page from their healthcare clients: not offering false hope and deconstructing the incident to learn from it.
At the recent CHIME Fall Forum in San Antonio Texas, I had the opportunity to sit down with Brenda Hodge, Chief Marketing Officer – Healthcare and Ed Rucinski, Senior Vice President of World Wide Healthcare Sales of Nuance to talk about the Petya outage and where the company is headed.
“The challenge we faced with Petya brought us all together as a company,” explained Ed. “When our systems went offline, the entire organization rallied together. We had engineers and support staff who slept at the office on couches and cots. We had developers who went with less than 2hrs of sleep for 4 days straight because they wanted to help clients and bring our systems back online as quickly as possible. We became a nameless and rank-less organization working towards a common goal.”
As the outage went from minutes to hours to days, Nuance resisted the temptation to offer false hope to its clients. Instead, the company opted to be truthful and transparent. Nuance sent emails and directly called clients to let them know they had suffered a cyber attack, that the full extent of the damage was not known and that they did not know when their systems would be back online. The company did, however, commit to providing regular updates and being available to answer questions and address concerns.
The following is an abbreviated excerpt from a Nuance communication posted online by one of its clients:
Nuance corporate systems were unfortunately affected by a global cyber attack today. We went into immediate security protocol by shutting down our hosted production systems and platforms. There is no update at this time as to when the accounts will be back online but we will be holding regular calls throughout the day and night to gain insight into the timeline for resolution and I will update you again when I have more info. We are sorry for the inconvenience this outage has caused and we are working diligently to get things back online.
Clinicians are coached never to give patients in crisis or their families false hope. They calmly explain what happened, state the facts and talk about potential next steps. They do not, however, say that “things will be alright”, even though they know that is what everyone desperately wants to hear. Nuance used this same protocol during the Petya outage.
The company also used protocols similar to those used following an adverse event.
Healthcare is complex and despite the best efforts and best intentions of care teams, errors occur. These errors are referred to as adverse events. Adverse events that impact patient safety or that cause actual harm to patients are thoroughly documented, deconstructed and analyzed by clinical leaders as well as risk managers. The lessons gleaned from these unfortunate events are captured and used to improve operations. The goal is to prevent or mitigate the impact of similar events in the future.
After their systems were fully restored, the Nuance team embarked on a thorough review of the incident – from technical procedures to client communication protocols.
“We learned a lot through this incident” says Hodge. “We got a first-hand education on how sophisticated malware has become. We’ve gone from viruses to malware to ransomware to coordinated nation-state attacks. That’s what Petya really is – a coordinated attack on company infrastructure. Now that we have been through this type of attack, we have put in new processes and technologies to prevent similar attacks in the future. Most importantly we have made investments in improving our response to these types of attacks.”
Nuance has gone one step further. They have committed to sharing their painful lessons learned with other companies and healthcare institutions. “Like it or not, we are all in this together”, continued Hodge. “The Petya attack came on the heels of the WannaCry ransomware attack that impacted many of our healthcare clients – so there was a lot of empathy from our clients. In fact this whole incident has created a sense of solidarity in the healthcare technology community. Cyber attacks are not going to stop and we need to come together as an industry so that we are as prepared as we can be for the next one.”
“It’s unfortunate that it took an incident like this to show us what we are made of,” says Rucinski. “We had executives making coffee and fetching lunch for the support teams. We had leaders offering to run errands for staff because they knew they were too tired to keep up with those types of things. In the end we found out we truly embody the values and principles that we have hanging on posters around the office.”