Fun with ransomware
I mentioned that in between the Pocketables move, trying to remote school my kids, big birthday for the wife (tomorrow,) that I was also dealing with a client’s ransomware infection. thought I’d tell the story of this so you have an idea of what my dire warnings to you coming up in the next few days will be.
This is a potentially altered timeline as I’m doing most of this from memory, and a slightly altered story as clients not particularly wanting to say “I clicked on ransomware!” as well as I don’t want to advertise to the ransomware people that the only people who might pay are currently down with Covid-19 as that story sounds like I’m just trying to buy time.
TL;DR – very long IT story, a squirrel blows up and takes out power to a building, the usual blah blah blah IT story.
Side note – we don’t actually know yet that they clicked ransomware. Could have been an exploit. Not blaming the client.
The event
Got a call from a client that all his icons had turned white and had been renamed. I told him to unplug the computer immediately. Yank it from the wall. He took that as choose shut down and told me this about two minutes into what would be a massively destructive run.
It was a Dharma-variant, encrypted the files, encrypted all the files his computer could see on the network, and even encrypted a computer he had admin access on but was not on in the same subnet. Basically about 218,000 files and 27 years of work were in various stages of destroyed. No, none of the Dharma decrypt apps have the decryption key. Yes, we tried the RannohDecryptor tool.
The malware also turned off file history, deleted backups, killed the backup software, all sorts of fun stuff. The sort of fun stuff we keep Veeam around for and nightly backups on site and off site.
As one of the workstation backups was old, I was asked to contact the ransomware people and see what exactly the demands were. I didn’t hear back from them for a day but when I did the demands were 1 bitcoin (about $10,500 at time of demand,) or all the data was going to be packaged up and sold on the dark web.
Someone’s going to really enjoy database from a church, 40,000 bitmaps, some emails from 1990s and a bunch of excel spreadsheets. Actually maybe they will. The church data however was something that was not backed up as it was sitting on a desktop of a computer that was essentially a spare and the IT guy did not know it was several years of data they needed, on a spinning hard drive, in the corner.
Tell your contracted IT people what that computer in the corner is for especially when it’s being used for a company/church not at all affiliated with your business.
And so it goes…
So I waited to forward the demands to the church as the contact was at his father’s funeral out of state. Yeah, seriously.
Ransomware people contacted me telling me the longer I waited the higher the price was going to be. My client said no, but the other data was another thing and I was not giving a total negative answer at that point.
I’d already restored the entire company servers to 10pm the night prior with maybe 7 clicks (Veeam for the win,) and was creating two new desktop machines as we do not re-use an infected operating system, ever. Don’t do it kids.
It was decided the setup was overly complicated and it was time to simplify it. Client’s system was running on a VMWare server with some ancient applications and servers – talking instances of virtualized Windows Servers from 2003-2019 because well, nothing other than the workstations were connected to the internet, some ancient never-updated custom applications, etc. Literally some of the apps there was no ability to go beyond Windows Server 2003 and they still get use occasionally.
They were never an exploit target from outside, but it was time to stop using them. They’d been lucky with security through obscurity and no internet access but something was going to come along. I really should have waited in hindsight. Just seemed like something I could do while I was making sure this never happened again.
Copied contents of the shares to the 2019 virtual server after resizing the disk image, remapped things, and Veeam started doing a backup of 500+ new gigs of data for the first time. As I’d resized the disks I got to wait 91 hours for the initial backup to stream off site to a remote location. Wasn’t touching anything without an off site in place. They’d been saved once, I was not pressing luck.
And then it poured…
Contact got in and was immediately quarantined for Covid-19 with symptoms, exposure to someone who tested positive, etc. Ransomware people contacted me again, I have no answer nor auth to tell them anything as I literally don’t have a contact. Well, I did, but I’m not bugging someone with a 103 temp back from their parent’s funeral that the pastor covid’d out the day prior.
The initial new file server location backup had completed but what I did not know was that that backup had started with 215 gigs left to copy. So there were 215 additional gigs to move out for the near TB of data.
A rogue squirrel 15 miles away got blown up in a power substation and took out power for hours. This doesn’t generally happen. This was not part of the DR data recovery plan. Building exploding part of plan, no power to restore in the recovery area, no.
While the backup server was busy doing a reverse incremental backup with 215 additional gigs the power went out. Not normally an issue as the two UPSs pick up and signal to the servers to shut down when power reaches about 30%. Evidently an issue this time just because it just had to be. Reverse incremental kept chugging along.
Shutdown server request did not come in time and the virtualized copy of the infected machine from pre-infection in instant machine restore disappeared, and the Veeam server went down hard while still operating. (Maybe the battery failed, no idea still)
I came in and pressed a few power buttons on Sunday. The virtualized infected machine backup (from pre-infection,) was missing, the Veeam server console refused to open. Scandisk, SFC /Scannow, and RAID all checked out. Windows error messages indicated an XML problem with dotnet and Veeam, and a kernelbase.dll issue if I remember correctly. Attempted to reinstall no dice dotnet was working from all I could tell. Diagnostics on dotnet were fine.
Veeam services running, it looked like it might be running a backup, but I could no longer see anything and needed that instant workstation restore and it was listing as unavailable on the VMWare server. Head to desk. I did the thing I never want to ever have to do – I opened a priority 1 ticket with Veeam.
20 minutes later I get back an email to rename C:\Users\Administrator\AppData\Local\Veeam_Software_Group_GmbH\veeam.backup.shell.exe_Url_hu1utqnj52thvmhrg5kie2bl15o22i22\10.0.0.0\user.config to something else. I think I named it user.config_old
Bam Veeam works again yay!
power goes out boo!
Trip back to the clients to find out the power didn’t go out and everything appears fine – uncertain noises!
Veeam pops up and tells me that CBT (or something,) is bad on that particular backup and it needs to do a complete re-do… sure, it shut down during a reverse-incremental full build if I remember correctly… 14 hours later it’s copied 200 of the 280 new gigs off site (it’s slow, only the new stuff moves) and the remote system goes down because of course it does.
One last hope
I went through several ransomware recovery guides and many mentioned EaseUS as a recovery option that might work. EaseUS asks me all the time if I want to test their product and as I’ve never been in a situation that needed or nor is Pocketables a particularly data-recovery oriented blog it’s basically been “I’ll contact you if I need you.” I needed them.
Unfortunately all my EaseUS PR contacts there were on an 8-day holiday and as such forget about me getting a free copy to try out for my client.
So client bought it for me, I installed and ran it with the infected SSD as a secondary drive and surprise surprise all his deleted and renamed files showed in two places, unfortunately they’d all been encrypted and anything recovered over to the new drive was just garbage. Not going to say this was EaseUS’s issue, it was up against data left from a program designed to destroy data, that turned off data recovery options, encrypted data, deleted the encrypted data, saved new data, and as far as I know filled the SSD before calling it a day. I don’t think there was anything left on that. I’ll be trying it on the spinner drive that was not infected but was accessed and encrypted later.
What else could happen?
So while I was working on the backup appliance and checking the logs I discovered that for the past 4 days there had been a series of security failures. Several thousand. All RDP related. Le sigh. 5+ year odd port mapping into the network for RDP had been discovered during the ransomware timeframe, because of course it had. Not much of an issue but they were throwing a dictionary at it.
Lesson kids: audit your client’s IT surface attack profile regularly, especially if you didn’t set it up.
Points of failure / things I learned
It turns out that the spare machine with the church data did have Veeam Agent on it, however the license did not work or had stopped working for some reason and the backups were quite old. No emails were generated that I can tell indicating that the product was not functioning, and nothing was presented to the user ever to say “your backups are not working and you’re a billion days out of date”. Or maybe it was and they simply didn’t tell me.
The assumption was that things were working and nothing told the user they weren’t or forced them to give me a call.
EaseUS recovery, I’m not going to say it failed because this wasn’t an accidentally deleted item and it wasn’t a drive failure, but yeah. We’ll see on the church computer. Don’t expect it to handle malware encryption regardless of how many articles point to it. If you can’t preview a JPG or DOC it shows, it’s not gonna work.
I had them install MalwareBytes and offlined the infected machine, it recognized the infection but was unable to clean it. I had to use two or three different things and edit the registry manually to play around on the infected offline machine. I’m actually unimpressed with MWB for the first time ever – charging to clean, and then not cleaning. Yeah, no free cleaning any more.
I had several articles that pointed me to a Kaspersky cleaning/recovery tool for this specific malware and nope, did not work. Ah well. most of them mentioned the products above as well.
Discovered BackBlaze per Daniel’s suggestion and deployed it on a couple of the machines – it’s a $6 a month machine backup to the cloud. Won’t work with servers, but those are all covered by Veeam at this site, and the workstations were all supposed to be. We’re throwing that in as all the things that went wrong in a week I no longer trust data only on one backup platform.
Evidently many ransomware recovery companies pay the ransom people and charge you extra so that when you/they get stiffed (evidently 13% of the time,) they’ve got a refund.
Seriously, check your backups
Even paying for licenses, backup apps can offline and not tell you bad things are happening. Do you know your backup is working? Check it. Set a day to check it. Sometimes the just set it and forget it option forgets it.
Are you backing up to a connected drive? Unplug it after the backup is done or guess what’s going to happen.
Are you sure that what’s being backed up to the cloud is what you think it is? Go to someone else’s computer or use your phone and verify a file or two.
Thinking about Veeam? Honestly other than the endpoint backup conking out (still don’t know why,) and the power outage fiasco it’s been mostly a pleasure. Most of the frustration I have here deals from trying to get 1tb through a VPN connection between two fiber connected points that barely is functioning at 10mbit sustained these days. Literally 5 days at this point waiting for this slow VPN connection (not Veeam’s fault,) to push data. If anyone has a point to point non-VPN Veeam walkthrough I’d appreciate it.
Thinking about EaseUS? If so just remember if you can’t preview the file (such as a JPG,) the data is corrupted or encrypted and paying for it is probably not going to help.
Thinking about MalwareBytes for a cleanup? Honestly disappointed with its failures this round. I mean as a thing it’s generally great, but having to subscribe to get a clean, and doing this on a machine that it knows is infected, seems to be a bad idea all around. Also we paid for a license for the client, it didn’t clean… first SMH at MWB.
I have no idea the condition of the squirrel, but I assume crispy.
This story of data recovery is what happened WITH backups and the ability to restore them instantly with a network appliance in building and off-site storage, SonicWall perimeter threat detection, McAfee AV evidently on the infected computer (I told them just use Defender and don’t install AV because every infected machine I’ve ever seen had antivirus software on it).
All’s backed up and recovered, except for the non-client/church, they’re probably screwed.
