Critical Infrastructure Security , Cybercrime , Cybercrime as-a-service
The Ransomware Files, Episode 6: Kaseya and REvilRansomware Gang Used Zero-Day Flaws for High-Profile Attack
The REvil ransomware gang's attack against the U.S. software company Kaseya in 2021 is not only among the largest ransomware attacks of all time, it's also one of the most intriguing.
It involves the use of zero-day software vulnerabilities known only to a handful of people, a race between attackers trying to snare ransom payments and defenders developing a patch, and a secret operation that hacked back against the REvil hackers. And in the end, a rare action happened: Someone was actually arrested.
This episode of "The Ransomware Files" talks to those who had a role in this incredible event. It also coincides with the release of new technical information about the software vulnerabilities exploited by the ransomware gang, which were found by the Dutch Institute for Vulnerability Disclosure, or DIVD.
REvil managed to exploit zero-day vulnerabilities in the Virtual Systems Administrator, which is remote management software made by Kaseya and widely used by managed service providers. The vulnerabilities allowed the group to spread its ransomware, which was disguised as a software update.
DIVD had warned Kaseya of the vulnerabilities in April, but REvil also discovered them, says Frank Breedijk, manager of DIVD's Computer Security Incident Response Team. Breedijk and DIVD's chairman, Victor Gevers, felt they had lost the race with the attackers.
"We were in this marathon to fix software that had quite a bit of technical debt in it," Breedijk says. "And then with the finish line in sight, on your right-hand side, all of a sudden comes Usain Bolt, passes you, flips you the bird and ransoms a whole bunch of systems."
The attack resulted in more than 1,500 organizations becoming infected with ransomware. Robert Cioffi is the founder of Progressive Computing, which is a New York-based managed service provider that used Kaseya's VSA to deliver services to its clients.
Cioffi feared he might lose his business after speaking with a colleague on the day of the attack in July 2021. All 80 of his customers were infected.
"I couldn't comprehend the words coming out of his mouth - that all of our customers were ransomwared," Cioffi says. "It just didn't make sense to me. What? How is it that everyone is ransomwared?"
There are other twists and turns. The FBI and its law enforcement partners hacked back at the hackers, snatching a universal decryption key. And after the REvil gang went dark for good, prosecutors announced the arrest of a Ukrainian man, Yaroslav Vasinskyi, for the attack against Kaseya. Vasinskyi is now awaiting trial in Texas (see: US Nabs Alleged Ransomware Operators - One Tied to Kaseya).
"The Ransomware Files" is a podcast miniseries available on Spotify, Apple Podcasts, Google, Audible, Stitcher and more. I'm speaking with those who have navigated their way through a ransomware incident to learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, and it's important to share the lessons.
If you enjoyed this episode of "The Ransomware Files," please follow it on a podcast platform and leave a review. Also, the show has a Twitter handle, @ransomwarefiles, that tweets news and happenings about ransomware.
If you would like to participate in this project and tell the information security community about your organization's brush with ransomware, please get in touch with me at firstname.lastname@example.org or direct message me here on Twitter. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, is no longer a threat.
Speakers: Robert Cioffi, Founder, Progressive Computing; Frank Breedijk, Manager, CSIRT, DIVD; Victor Gevers, Chairman, DIVD; Jason Manar, Chief Information Security Officer, Kaseya; Jon DiMaggio, Chief Security Strategist, Analyst1; John Hammond, Senior Security Researcher, Huntress; Espen Johansen, Security Director, Visma; Adrian Stanila, Senior Information Security Researcher, Visma; George Zamfir, Security Analyst, Visma; Jeremy Kirk, Executive Editor, Information Security Media Group.
Production Coordinator: Rashmi Ramesh.
The Ransomware Files theme song by Chris Gilbert/© Ordinary Weirdos Music.
Music by Uppbeat andPodcastmusic.com.
- Bisend, What is Classic ASP?, Jan. 28, 2019;
- Data Breach Today, REvil Revelations: Law Enforcement Behind Disruptions, Oct. 22, 2021;
- Double Pulsar, Kaseya Supply Chain Attack Delivers Mass Ransomware Event to US Companies, July 3, 2021;
- Dutch Institute for Vulnerability Disclosure, Why We are Only Disclosing Limited Details on the Kaseya Vulnerabilities, July 7, 2021;
- Huntress, The Hunt to Find Origins of Kaseya's VSA Mass Ransomware Incident, July 20, 2021;
- Reuters, Governments Turn Tables on Ransomware Gang REvil by Pushing it Offline, Oct. 22, 2021;
- The Record, Kaseya: More Than 1,500 Downstream Businesses Impacted by Ransomware Attack, July 6, 2021;
- Visma, Software Vendor Kaseya Exposed to Global Cyberattack, Affecting Retail Trade, July 3, 2021.
- Adrian Stanila, Kaseya War Stories, Nov. 22, 2021;
- Allan Liska, Ransomware: Understand. Prevent. Recover, Oct. 28, 2021.
- Huntress Labs, Reddit post: Critical Ransomware Incident in Progress, July 3, 2021;
- Kevin Beaumont, Twitter post, July 5, 2021.
Merrick Garland: Cybercrime takes many forms, one of which is ransomware. In ransomware attacks, transnational cyber criminals use malicious software to hold digital systems hostage and demand a ransom. These attacks have targeted our critical infrastructure, law enforcement agencies, hospitals, schools, municipalities and businesses of all sizes. Today, we are announcing that we are bringing to justice an alleged perpetrator of significant wide-reaching ransomware attack. On July 2, multinational information software company Kaseya and its customers were attacked by one of the most prolific strains of ransomware, known as REvil or Sodinokibi.
Jeremy Kirk: That is the voice of U.S. Attorney General Merrick Garland. He's the most senior law enforcement official in the United States. In November 2022, he had an extraordinary announcement in the fight against ransomware, one of the greatest crime waves to ever affect the internet.
A 22-year-old Ukrainian man named Yaroslav Vasinskyi was arrested as he crossed into Poland. Vasinskyi had been indicted three months prior, on charges of perpetrating one of the most devastating and far-reaching ransomware attacks. The attack used software vulnerabilities in a remote management software made by an American company called Kaseya. Vasinskyi was allegedly involved with a prolifically destructive ransomware gang that called itself REvil, which is short for Ransomware Evil.
The incident is not only amongst the largest ransomware attacks of all time, but it's also one of the most intriguing. It involves the use of zero-day software vulnerabilities that was known to only to a handful of people. It involves a race between attackers to try to snare a ransom payment and defenders developing a patch. It involves a secret operation that hacked back against the REvil hackers. And that's just before we get to the sweeping effects that this attack had on its victims. Around 1,500 were affected. Among those were Robert Cioffi. He's the founder of Progressive Computing, which is a New York-based managed service provider - or MSP. MSPs manage IT systems on the behalf of other organizations. Robert's company used a type of software developed by Kaseya, called the Virtual Systems Administrator, to do that. All 80 of his clients, which were mostly small businesses, were infected. On the day of the attack - July 2, 2021 - Robert wondered if his business of nearly 30 years was going to survive.
Robert Cioffi: You know when they say 'your life flashes before your eyes'? That's what happened to me. On July 2 of 2021, 28-and-a-half years of my life and my business life flashed before my eyes. I had thoughts about what my future might look like, that were in very dark recesses of people's minds. All the things that you think about - that nightmare scenario - suddenly became a potential reality for me, and quite frankly, scared the crap out of me.
Kirk: We're going to hear more from Robert later. This episode is a companion to Episode 5 of The Ransomware Files, which focused on REvil's attack against an MSP in Texas. That attack resulted in 23 Texan cities being infected. Both the Texas and Kaseya incidents share unique characteristics: both were conducted by REvil affiliates, both are what many would describe as supply chain attacks, and both resulted in a rare event: successful law enforcement actions.
This episode is going to interview key people around the Kaseya incident and even Kaseya itself. Also, this episode will have new technical information about the Kaseya vulnerabilities from the Dutch research group that discovered the problems.
This is The Ransomware Files. I'm Jeremy Kirk.
In this podcast mini-series, I'm speaking with those who have navigated their way through a ransomware incident and learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, but it's important to share the lessons.
Some say 2016 was the year of ransomware. Although ransomware had been around for decades, that year is when ransomware gangs began to really hone their skills. But you could also say that every year after 2016 has been the year of ransomware. The groups have continually improved their file-encrypting malware, developed business models that expanded their reach and in the end, compromised an ever-increasing number of victims. Also, there's the money: It's a billion-plus dollar industry.
Ransomware gangs are looking to infiltrate managed service providers, or MSPs. MSPs fill a critical gap for organizations that don't have the right amount of in-house IT skill. MSPs help with patching applications, installing applications, making sure email works properly. To do that cost effectively, MSPs use remote access software to do those tasks from afar. Ransomware groups saw this as an opportunity: compromise an MSP, and you could strike gold by potentially infecting all of their clients. Organizations such as the U.S. Cybersecurity and Infrastructure Security Agency had long warned that state-sponsored hackers were hunting for vulnerable MSPs for cyberespionage and intellectual property theft. Then the ransomware gangs joined in.
There isn't a huge number of remote monitoring and management software vendors for MSPs. But Kaseya is one of the most popular ones. It develops an application called the Virtual Systems Administrator. Its logo kind of looks an italicized version of the one for Visa's credit or debit cards, but without the "i."
The story of the Kaseya incident starts in April 2021. Hmmm, scratch that. The story of Kaseya starts in the days when applications were often coded in the programming language known as classic ASP, which is short for Active Server Pages. This was Microsoft's first server-side scripting language, and it was introduced in the 1990s. It revolutionized web applications and how web pages were built.
Kaseya's VSA was written in classic ASP. It's not a bad programming language per se, but it's just that time has long since passed it by. And this is the crucial part: it was just easy in classic ASP to make coding mistakes that led to SQL injection flaws, cross-site scripting issues and so on. Kaseya's VSA has since had a major overhaul. But in the couple of years leading up to the incident, cybercriminals and other groups had been paying close attention to vulnerabilities or potential vulnerabilities in remote management software.
But one of those groups was on the good side. The Dutch Institute for Vulnerability Disclosure - or DIVD - is a volunteer group of security researchers. It was founded by Victor Gevers. He a hacking legend. He could have taken over former President Donald Trump's Twitter account, not once but twice, after guessing Trump's weak passwords. Gevers and other volunteers with DIVD hunt for high-impact software vulnerabilities, responsibly report them to the appropriate vendor, and then hope they get fixed quickly. Their mission is simple - make the internet safer.
Frank Breedijk is the manager of DIVD's Computer Security Incident Response Team. He has been with DIVD since day two and knows the story of Kaseya inside out. One of DIVD's researchers, Wietse Boonstra, encountered Kaseya's VSA software last year while doing a pen test. So Wietse is what they call a penetration tester. Companies hire him to try to break into their networks so they can fix the problems they find. Here's Frank.
Frank Breedijk: During a pen test, a customer he encountered Kaseya - Kaseya VSA, I should say. He was running a pen test. He got that irking feeling that "you know what, it looks like there's more to find here," but he was running out of time. And Wietse being Wietse, he decided: "you know what, I'll take the software home or try to find a copy that I can download somewhere". So he installed it on a local server. When other people watch Netflix for fun, Wietse likes to fool around with software for fun. While he was sitting on the couch with his significant other, he was testing around and playing around with the software. And that's how we first encountered the vulnerabilities. And, yeah, it was a matter of "this is serious, we better contact Kaseya."
Kirk: DIVD found around seven issues, and one was particularly bad. It was an authentication bypass for the on-premises versions of VSA that were facing the internet. Now, that's a lot of jargon so to explain that a simpler way, many managed service providers used VSA to manage their clients' infrastructure. And many left that VSA server just on the internet. It meant that anyone in theory could just browse to the MSP's VSA server and see a login and password dialog box. An authentication bypass means you can get around that dialog without using a username and password. Next, Frank explains how this flaw worked. And to note, he and DIVD have never described it to this level of detail until now, but they believe enough time has passed that it is safe to do so.
Breedijk: One of the core problems in the authentication bypass we found was that it was hard to fix, because it was actually in the business logic. So there is a page you could download the Kaseya client from. After the Kaseya client was installed, it needed to authenticate itself to the Kaseya server. I'm disclosing details now that we haven't disclosed before… So the authentication bypass that we found: there was a page called DL dot ASP - I think the DL stands for download. On that DL dot ASP page, there was a link where you could download the executable for the Kaseya agent. If you downloaded the Kaseya agent and clicked install, it would generate an 'ini' file on disk. And that ini file actually contained the Kaseya user agent, Kaseya agent password and username. And with that password and username, you could also authenticate yourself and obtain assess valid session ID for anything on the web interface.
Kirk: DIVD found other issues too, including SQL injection flaws. Most affected not only the on-premises version of VSA, but also the software-as-a-service version, with the exception of the authentication bypass. DIVD promptly reported the flaws to Kaseya in April 2021. And as many security researchers know, reporting security flaws to organizations can be fraught with tension, and it doesn't always go well. But DIVD founder Victor Gevers says Kaseya was welcoming.
Victor Gevers: Wietse Boonstra started investigating the server and analyzing it locally. He found a few vulnerabilities very quickly. The most interesting part, for me, was that Kaseya was very open and it was easy to get in touch with them. The first meeting was a virtual one because of the pandemic. The first conversation went pretty well. We discussed each issue that needs to be fixed as soon as possible, and set up a safe way for Boonstra to share his research. p>
Kirk: In fact, Kaseya created user accounts for DIVD researchers so they could collaborate on Microsoft Teams. The researchers had video call with Kaseya's CTO. Frank says the reaction from Kaseya was instant and positive.
Breedijk: I've never seen this kind of collaboration and participation at this level. Because we're part of their organization, it meant that we could reach out to them a lot easier. Sometimes, as easy as saying, "hey, we found something new, or [that we found] this mitigation," and we could shoot off a message saying that were working on that. And they would bounce ideas off us as well. For instance, they'd discuss if something would work better one way or if it would be better to tackle it in a different way. I think it was important, because they had built up a lot of technical debt.
Kirk: Between April and July of 2021, Kaseya fixed some of the flaws but not all of them, including that all-powerful authentication bypass. There's always a nervous period of exposure between when a vulnerability is privately disclosed to an organization and when it is patched. Bug reports are usually highly confidential information because if the attackers get the information, they could use it. Little did anyone know, REvil was about to shock everybody.(Music transition)
July 2, 2021, was a Friday. And in the U.S, it was just ahead of the fourth of July weekend, and attackers love to ruin a good weekend. They rightly anticipated that defenders would start to relax a bit and maybe would be a bit less attentive than normal.
George Zamfir is a security analyst with Visma Group, which is a large IT services and consulting company headquartered in Oslo. George works from Sibiu, a city in central Romania. He's a former software developer who got into security, and he's pretty handy with the Python programming language. He ended up writing a tool that Visma now uses daily for threat intelligence. The tool didn't have a name, so someone suggested they call it Cyber George. Here’s George.
George Zamfir: I was like, "Damn, that's good. I'm going to stick with that." So, I'm cyber George. And we also have a tool called Cyber George.
Kirk:That Friday marked the last day of George's first week on duty shift with Visma, monitoring security alerts. He had a couple of small alerts earlier in the week. Then on Friday, he got a weird feeling while he was speaking to a colleague.
Zamfir: Around 9 pm, I was like, "I feel like there's a third incident coming. I can feel it." And the guy was like, "why, what's happening?" That was the point when everything started to get a bit weird. The alerts started to come in for things that should not trigger an alert. I started to see ransomware alerts.
Kirk:Visma had three servers running VSA in its data center, and it was used to service clients such as the Coop grocery store chain in Sweden and a pharmacy there called Apotek Hjärtat. Endpoint detection and response software running on those machines detected the attack and raised alerts of possible ransomware. Little did George know, it would be start of an absolutely huge incident.
Adrian Stanila is a senior information security researcher with Visma, who is also based in Romania.
Adrian Stanila: For George, it was baptism of fire because it was the his first on-duty week. It was the first incident, and I think it was along similar lines as NotPetya from some perspectives.
Kirk: George started looking around the internet for clues, to check if other people were seeing the same things he saw. He found something on Reddit. A security company called Huntress Labs had started a thread on a subreddit for MSPs. Huntress Labs is based in the state of Maryland in the U.S, and helps managed service providers with security. John Hammond is a senior security researcher with Huntress.
John Hammond: And that day, the weekend of July 4, things were starting off as they usually do. I had a couple alerts in the dashboard, a couple things that needed investigation, reporting, etc. Truthfully, I don't know what the match to start the fire looked like, but we noticed, after a couple of different reports of ransomware that kept popping up, that there seemed to be some sort of commonality. One report of ransomware on a partner came up, and then another, and then maybe a third - each of them seemed to be working and running in their computer network and their system with the Kaseya VSA software, using it as the remote monitoring and management provider. It took some of the head honchos and other team members to start to think "this looks like it might be a trend." Two data points might make a line, but when we start to have more here, maybe something's going on.
Kirk: John says there were enough odd things going on for Huntress CEO Kyle Hanslovan to reach out directly to Kaseya.
Hammond: We have some crazy screenshots of Kyle, our CEO, just dropping a note saying, "hey, can we get on the phone real quick? Can we just hop on a Zoom session? Because we have some critical things that we'd like to just chat about." And then, our numbers grew from three to eight of known affected hosts, then 15, then 30, and then more, until eventually it became the story that folks know today.
Kirk: Huntress published a really detailed thread on Reddit, which became almost kind of a North Star for other organizations seeking out information.
I was curious. Why Reddit? I love Reddit, but I just don't see that kind of stuff shared on Reddit usually, I guess?
Hammond: Oh, totally. I'm glad you asked, because you're right. Normally, it's not the most professional avenue to offer threat intelligence, nerdy code, syntax and indicators of compromise. But just as you mentioned, people like Reddit; people hang out on Reddit or a Discord community or a Slack community. But it's natural for managed service providers to be in that R/MSP subreddit. So we go where the fish are, right? We wanted to get this in front of as many people as possible.
Kirk: Meanwhile in New York, Robert Cioffi of Progressive Computing was looking forward to a nice long weekend off. He says the weather in New York was perfect.
Robert Cioffi: I'm focused on the weather, because as a New Yorker, as someone who lives in the northeast, we have some rough winters. And you really relish those bright sunny days. The mood was pretty jovial here, we're all looking forward to just disconnecting for three days.
Kirk: It was around lunchtime, and Robert says he was in his kitchen when he first found out something was going on.
Cioffi: It was around lunchtime because I was in my kitchen. When my director of operations came upstairs and was walking down the hall. I had a clear line of sight to see him out of the kitchen door, and I knew something was wrong. I knew something was dreadfully wrong, because he was pale white. I had commented to one of my team members who was sitting next to me, saying "I think somebody just died," because that was the look on Jay's face. Jay looked like he was coming to deliver some really bad news about somebody's death. And then, when I approached him to ask him what happened, he began to tell me that all of our customers were ransomwared, and that the phones were ringing off the hook. It even took me a few moments of questioning him because I was in disbelief myself. He was already in, you know, a few layers deep into that shock. And I was just starting to experience the first taste of that shock, because I couldn't comprehend the words coming out of his mouth, that all of our customer - all of our customers - were ransomwared. It just didn't make sense to me. What? How is it that everyone is ransomwared?
Kirk:At the time, Progressive Computing had 80 customers spread amongst 200 physical locations.
Cioffi: He just kept insisting the phones are ringing off the hook with every single account. And then he just started to name them. And I think that was the moment in which this really black cloud started to envelop me despite the beautiful weather. And despite all our great plans for that weekend, the mood just changed dramatically.
Kirk: REvil had just pulled off quite a caper. It had launched a fast and furious attack using the authentication bypass vulnerability and others. Some of the same issues that Victor Gevers and DIVD had reported to Kaseya had been discovered by the ransomware attackers. These were zero-day vulnerabilities that were supposed to be secret. So what was going on? There are several possible explanations. First, it's not unheard of for vulnerability researchers working separately to stumble across some of the same vulnerabilities at the same time. And REvil was a well-funded group. It had been hiring penetration testers and bug hunters to find new avenues to spread its ransomware. Another possibility is that perhaps Kaseya's systems had already been hacked, and that the hackers were inside its internal ticketing or bug reporting systems. A third possibility is that the DIVD - the bug hunters who reported the vulnerabilities in April to Kaseya - was hacked.
We don't really know which of these actually led to REvil finding the flaws. Here's Frank Breedjik.
Breedjik Yeah, so when we find out that REvil was exploiting the same thing that we were doing investigation on, it scared the living daylights out of us. Why? Well, we thought we were maybe leaking that data. I mean, you're holding a vulnerability that has this devastating effect, and then somebody else starts using and at first you don't know what they're using.
Kirk: The attackers had used the authentication bypass and other vulnerabilities to begin distributing a payload. The payload was basically a bogus update for the Kaseya agent, which turned out to be the ransomware that belonged to REvil. There were indications, however, that REvil pushed the attack quickly ahead on the fly. For example, they didn't delete volume shadow copies, which is a Windows backup mechanism. Intact shadow copies can help organizations recover faster, which of course ransomware groups want to prevent. REvil also did not exfiltrate data. Usually, REvil extracted sensitive data before encrypting it so it could release that data publicly. REvil often dumped the data of its victims on a blog it called its Happy Blog. But both of these non-actions were a sign that perhaps REvil knew that Kaseya was close to patching, and that it had hurry up with the attack and just hit as many MSPs as it could.
There were at least 50 MSPs that had vulnerable Kaseya instances that were exploited. But this kind of attack flows downstream. So it's not just those MSPs but their customers. Estimates of as many as 1,500 customers of those affected MSPs were infected. And we see from Robert of Progressive Computing that the attack against his VSA server affected 80 customers downstream. A lot of those downstream victims never really surfaced publicly. But some of the big ones did. The Coop grocery chain in Sweden shut down some 800 stores because their POS systems were infected. It also affected the POS systems for the Swedish pharmacy.
That's what makes Espen Johansen so mad about ransomware. Espen is the security director for Visma Group, and he works with Adrian Stanila and Cyber George. He says ransomware affects real people. In this incident, people couldn't pay for their medicine at the pharmacies in Sweden because of ransomware. They couldn’t use their local grocery store because of ransomware.
Espen Johansen: You can see the kind of effects this ransomware has. It's just a bunch of morons sitting there trying to make a buck from injecting some viruses and in a gamer-like existence, and then you have actually shopkeepers who have to close down their shops and pharmacies that cannot deliver medicines to their customers, which is of course the other end of such attacks.
Kirk: Frank says DIVD was in touch with Kaseya after the attack started, and he says company officials were initially shell shocked. DIVD told Kaseya in a typical direct Dutch way that they were screwed. Well, the word that Frank used starts with "f" and ends in "ed," but we don't need to say that here.
But Kaseya immediately did a few remarkable things. On the technical side, they shut down the software-as-a-service version of VSA. Although that version wasn't vulnerable to the authentication bypass, there were other vulnerabilities in the product. Then DIVD, Kaseya and other agencies began pushing the word to organizations still running on-premises VSA servers that were facing the internet needed to take it them offline immediately. And just a couple of days after the attack, the vast majority of vulnerable VSA instances were offline. And Kaseya embarked on a very open and transparent campaign to keep its customers and other affected parties in the loop about the incident. And it's important to note that even in an age when most organizations, at one time or another, have had a security incident, they're not always handled well. But Kaseya began publishing regular and highly detailed updates on its website. Kaseya's CEO, Fred Voccola, released video updates. They also called on the sales team back and tasked it with reaching out to every single one of the company's 35,000 or so customers - even those not affected by the attack. And in the background, Kaseya was doing something remarkable: it started working straight away with the FBI.
Jason Manar: The cooperation is a reflection of the company, direct reflection of the CEO and the leadership within the Kaseya. So you have people that took unprecedented steps that during an incident that, quite frankly, I hadn't seen before, which is one of the reasons I joined them.
Kirk: That is Jason Manar. Jason was a cybercrime supervisory special agent with the FBI in Miami and led the investigation into the Kaseya incident. Three months after the attack, Kaseya actually hired him as its chief information security officer. During his time as an FBI agent, Jason saw that a lot of organizations didn't handle breaches or ransomware incidents with the same openness. Here, Jason describes how Kaseya promptly posted information about the incident in a long, rolling blog post.
Manar: What was so striking was that I wasn't even with the company. I was with the FBI. And I had their page up, because I wanted to see what information they were providing the general public, and literally within minutes of a conversation that I would have with them, they would update that page. And I was just like, "I've never seen this before, ever." Usually, it's a generic sentence, marketing to everybody in the company, and a lot of times it is straight up denial. And not only that, but especially when you worked for law enforcement, as I did in the FBI, you get people that wouldn't provide information to you to help with the investigation. And that was not that case with Kaseya at all. They wanted to make sure that the FBI got information that was directly attributed to the attackers, so that they could take action. And of course, we now know that with that information, they were able to take action.
Kirk: So there are several crazy twists to the Kaseya story. Here's one of them. Three weeks after the attack, Kaseya announced it had a universal decryption key. Like that. Magic. Somehow, the FBI and its foreign law enforcement partners managed to get a universal decryption key that would unlock all of the computers affected by REvil's ransomware. The technical details about how this key was obtained are unknown, but we generally know that law enforcement got inside REvil's servers. But I knew Jason knew more. And I knew he probably wasn't going to tell me.
We know from the record that something really extraordinary happens. The key is obtained - the key that will decrypt all the systems affected by this ransomware campaign. So how does that happen?
Manar: Well, I can say, because I was on the other side. It's very interesting how that happens. And I wish I could disclose more to you. But I will leave that for the FBI to share what it wants to share about that process.
Kirk: OK, I tried. To be honest, the continuing mystery of precisely how the universal decryption key was obtained is probably more alluring than the actual answer. But we can put together a likely scenario here. Ransomware gangs often have terrible operational security. They make mistakes when setting up their systems, which can allow researchers and law enforcement to get inside their servers. As an example, in February 2022, the Conti ransomware gang saw years of chat logs publicly released, providing researchers a rich trove of data on the inner workings of this criminal gang. At least two security companies, and possibly law enforcement, had been inside Conti's private Jabber server, watching cybercriminals chat for years. That kind of material is pretty useful for cybercrime analysts like Jon DiMaggio. He's the chief security strategist with Analyst1, which is a cybersecurity company based in Virginia. He studies and analyzes cybercrime and attacks, a skill that come from an interesting past.
I did have a question about your background. So when I describe you, if I described you as a former spy - is that taking it too far?
Jon DiMaggio: I love it. I love it if you want to say that. I would never say that about myself. But my kids will think that's so freaking cool. So I don't have any problem with it.
Kirk: Seriously though, Jon used to work in signals intelligence for a U.S. government agency that he can't name. Earlier this year, Jon wrote the definitive history of the REvil ransomware gang. It reads like a cybercrime mafia novel: there's extortion, there's loads of cash, there's betrayal, there are arrests in Russia and elsewhere and ultimately, the downfall of the group.
Jon says that leading up to the Kaseya incident, the U.S. and other governments were increasingly getting frustrated with this flagrant group. In May, a former REvil affiliate known as DarkSide ransomwared the energy company Colonial Pipeline. The attack resulted in fuel shortages because although the ransomware didn't affect the pipeline itself, the pipeline was shut down as a precaution. And in the same month, a REvil affiliate struck JBS Foods, one of the largest meat producers in the world, and disrupted production. These weren't trivial attacks. They were striking at national infrastructure related to energy and food production, which quickly raises the hair on everyone's necks. Plus, they were just flagrant, pompous jerks. The group bragged about its ransomware, which were upwards of $200 million. Members of the group gave press interviews. Its swagger was huge, and Jon says the group grew increasingly careless.
DiMaggio: As they got bigger, it just seemed like they were less alert, they didn't monitor things as well. And they just got sloppy and their ego seemed to really overtake their operational security. I think that we know, when you have discipline and don't let yourself get to a point where you rush and make mistakes; or have a few drinks and jump online to do work [this is work to them]… you just you get sloppy. I really do think that's what this was - a combination of ego, and just getting too comfortable believing that they would never ever get caught.
Kirk: Law enforcement capitalized on whatever mistake REvil made and snatched the universal key. We know from the timeline that the FBI and its partners obtained the key just within a day or two after the attack. So that means they were already in position before the attack happened and were ready to strike. But law enforcement didn't give the key to Kaseya immediately. It was only released three weeks later, which was a decision that proved to be quite controversial. In that interim, there were companies that were paying individual ransoms to REvil to get the key. And there were also suggestions that perhaps Kaseya should cough up $70 million REvil was asking for in exchange for that universal key, which would bailout all of those 1,500 victims. We found out later why the key was held back. At a congressional hearing, FBI Director Christopher Wray said that the agency was working an investigation into REvil.
(Sound clips from Christopher Wray)
Wray said that bureau didn't want to tip off attackers that law enforcement had gained some access to their infrastructure. It was undoubtedly a tough call. I asked Robert Cioffi of Progressive Computing how much would it have helped to have the key earlier, and if that trade off was worth it.
Cioffi: Yeah, I think it's an excellent and fair question. The FBI had that decryption key within a matter of days of the attack - would that have helped us and all of our customers? I can't even begin to illustrate the emotional, psychological and impact that it had on our relationships with customers, the goodwill that was destroyed by this attack, the questions that came up, and the hundreds of thousands of dollars in expenses that came that arose out of this attack. It would have saved all of us an immense amount of grief. Now, all that being said, from a guy who is at ground zero in this and that was not only professionally, but even personally affected by this. Do I support the FBI, their decision to withhold that information if they felt that they could have maybe captured some criminals? I'd say yes. If I had my video camera on, you'd see an American flag behind me. And I didn't hang that there just because I had no place else to put it. But I really wanted people to see, especially during COVID times, that I'm a proud American. And even though my parents are immigrants from a different country, I mean, that's why they came here - this country has offered me and my business partner, our families and our employees so much that I think in this case, we had to take it on the chin for the good of our ability of law enforcement to go after these guys. So I'm the first one to wrap them the flag around me and say, "We had to do what was necessary." I'm sure if you'd asked me at that moment in time, we can either have the decryption key or we can go after the bad guys, I'd probably say give me the key. But I certainly don't harbor any ill will. I'm sure it was a tough decision that was made at some very high levels within the FBI. And if they felt that was the right thing to do, then I'm going to support that 100%.
Kirk: But we know it ultimately worked. You heard Attorney General Merrick Garland at the beginning of this episode. In March 2022, Yaroslav Vasinski was extradited from Poland to the United States and will face charges for his alleged involvement in the attack against Kaseya. Vasinski is now awaiting trial in Texas.(Music transition)
As this episode is released, the Kaseya incident is still pretty fresh, as it occurred just around eight months ago. It's still sensitive, and the reflections of those involved are still vivid and somewhat raw. Cyber George of Visma says it was one of the best experiences of his life. He doesn't mean that he'd wish it to happen again, of course. But from a professional standpoint, he was at ground zero for one of the biggest ransomware incidents. He says figuring what was going was like hunting Moby Dick.
Robert of Progressive Computing started to choke up when he told me about a man from middle of Iowa who flew out to New York to help his company and his company's clients recover. The man was a complete stranger to Robert but had heard through one of Robert's friends that help was needed. And the person in Iowa wasn't the only one who jumped up to help. Robert says that it’s a sign of the closeness of the MSP community, and how that bond can beat cybercrime. Roberts says that also helps him tamp down the anger that the incident can still stir up.
Cioffi There is that element of, I'll say that anger that I'm always just trying to fuel into something more positive. Instead of gritting my teeth and wanting to meet that guy when he gets extradited to the US. I told the FBI, by the way, "if you guys have a little trouble making something stick, just call me out. Put me in a room with the guy alone and go get doughnuts or something. I'll take care of him."
Kirk: You know you have Italian heritage right? So don't mess with that.
Cioffi: Don't Mess With Texas and don't piss off in New York Italian unless you want to end up someplace you shouldn't go. I'm joking, of course. But yeah, there is that passion and fire in me that I'm not going to take this lying down and I'm not going to allow my community to suffer the way we suffered.(Music transition)
Kirk: But there's one particular part of this story that really stood out to me. Victor Gevers of DIVD wrote this on Twitter on July 3, 2021, just one day after the attack: "Somewhere this year, the DIVD will share a story about how we nearly prevented an enormous supply chain ransomware attack which potentially led to the single largest ransomware spree in history, and failed." I asked Frank why they thought they failed.
Breedijk: Because people got ransomed. It felt like, together with Kaseya, we were in this marathon to fix software that had quite a bit of technical depth in it. And we were nearing the finish line, four out of the five bugs were fixed. We were going to scan the internet, make sure that everybody had their patches installed. And then when you near the finish line, on your right hand side, all of a sudden comes Usain Bolt, flips you the bird, ransoms a whole bunch of systems and beats you to the finish line. So we failed in the sense that we weren't able to prevent this. And for us, the dream outcome would have been to sit down with Kaseya, have a patch, have everybody install the patch and theb publish a story on how we found this huge potential hole that could have could have been devastating. If we had said it would have shut down half of the supermarkets in Denmark, people would have laughed at us, because they would have thought we were probably overstating our own importance. But yeah, in the end, it did happen.
Kirk:I think DIVD is being too harsh on themselves. And hey, there was a good outcome to this. Yaroslav Vasinski is going to be prosecuted. Even Russia, in January 2022, rounded up more than a dozen alleged member of REvil in a surprising law enforcement action. Now, of course, we don't where that going to go these days in light of world events, and it's somewhat doubtful it's going to go very far.
Robert of Progressive Computing says all of his customers eventually recovered because they all had good backups, but it was a heck of a lot of work. And Espen says Visma helped the Coop grocery chain get back on its feet. Some stores had to be visited in person to install new images for the POS systems, and Espen says a lot of people spent their entire summer vacations working on that. To be sure, disaster recovery isn't for the meek.
But the REvil gang is no more. Sure, its affiliates and members may have joined other groups now. But maybe that's a sign that the increased focus on ransomware and new strategies to fight ransomware are working. It's going to be a long slog, of course, but it's OK to notch a win now and again.
This episode of The Ransomware Files was written, researched, edited and produced by me, Jeremy Kirk. The production coordinator is Rashmi Ramesh. The Ransomware Files theme song is by Chris Gilbert of Ordinary Weirdos Music.
If you enjoyed this episode of The Ransomware Files, please share it and leave a review. It will help me keep this project going. Also if you haven't already, go back and listen to Episode 5, which is about REvil and its attack on Texas. Also, the series has its own Twitter handle @ransomwarefiles, which tweets news and happenings about ransomware. And I'm on Twitter @jeremy_kirk. If you would like to participate in this project or have an idea for it, please get in touch with me. My direct messages are open on Twitter and I'm easy to find on LinkedIn. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.