Welcome to episode 396 of the Microsoft Cloud IT Pro podcast recorded live on 02/21/2025. This is a show about Microsoft three sixty five and Azure Azure from the perspective of IT pros and end users, where we discuss a topic or recent news and how it relates to you. For today's episode, Ben walks Scott through a ransomware attack on an on premises VMware environment. As we walk through the attack, we also talk through lessons learned, things not to do, and ways to help mitigate such an attack.
So here we are. We did no planning, Scott, So we're telling stories. Should we tell the story today? I think you have a fun story to tell this week. I I always think it's interesting to eye the more material and conceptual things back to the real world. So today, you've got a nice fun story. You know, I was trying to think about, like, what could we title this one? Because, you know, it's fun to kinda game the titles in the beginning
and and see where they would go. And I kinda landed on just lessons from a real life ransomware attack. And we'll have we'll and take us through a real life ransomware attack and some things that went wrong at client that you were working with. And we won't name names or anything like that, but I think it's a good opportunity to talk through potentially what that client of yours could have done better, especially
looking back in time. And, you know, I think there's some good lessons in there around, like, perceived PCO of things, like having multiple copies and all that good stuff that we can get. So why don't you start us out at the beginning? Yeah. So this is absolutely one of those boat one of those scenarios. And like you said, names, people, places, things will be left out to protect said people. I'm also on the fence guide. I'm always like, do I wanna call them
a client? Like, I've been working for them for two weeks. They were not a client before two weeks because I don't wanna be like, hey, Ben, why didn't you protect these people sooner? That type of scenario. But they are a client. It is a something I've been working on for two weeks. So this was, I think it was maybe a week and a half ago, something I was, I can turn this into a whole narrative too. So, yeah, I was actually talking to a partner I have and we were talking
about another project. He was like, Ben gotta go. Cyber incident just came in. So it was like, okay, go let me know if you need anything. Didn't think much of it. I don't always get pulled into these because again, I deal primarily Microsoft three sixty five Azure, but got pulled in a little later, talked to them. They thought everybody was out. They definitely had been compromised, but thought they caught it early enough, pulled network access to on prem hardware soon enough,
all of that. So chatted with him for a couple more hours. Great. Was sitting in my office. I should have gone to bed. One lesson learned, Scott. After that, just go to bed and turn off the cell phone. Always good. Either recommended. Yeah. So I was sitting in the office. 11:30 at night, I get a text message. Hey, Ben. Are you still awake? Dilemma. Do I answer do I respond or do I pretend I'm asleep? Yeah. I'm awake. All of the VMs in this environment just
shut down. We lost all of them. And it's not a call you wanna wake up to. No. This is not what you wanna hear. So okay. Let me jump on the call. Lo and behold, these attackers had gotten into the network, And this is where the first lesson is, Scott. This was not an attack I had necessarily heard of before. In this particular scenario, there were some unpatched networks, firewalls in this environment, and there was a certain vulnerability. And, again, I'm not sure of the vulnerability.
I'm not gonna name brands or any of that, but it allowed them to actually create a based on my understanding of this, it allowed them to create a VPN connection to the firewall to get inside the network. So this was not a user compromise. This was not somebody clicking on a phishing link and credentials getting accessed. It was, oh, here's a vulnerability. Let's establish a direct connection into the internal network through the VPN
and see what we can find. So first rule, servers are not the only things you should be patching and updating, and the only things with vulnerabilities are workstations. Probably not. You know, gen generally, firewalls are an important thing to patch. Your firewalls, your switches, let's just say your entire infrastructure And end to end. Keep these updated. So once they're on the network, now it's like, let's go
see what we can find. They managed to get their way into the ESX servers that were hosting all of their on prem VMs and encrypt everything. Encrypt drives, encrypt configuration files, Everything was encrypted, and this is why said servers went down. They were just no longer in existence, or they weren't in existence. They were just no longer accessible anymore. Oops. They went away. Oops. Right? So first thing, right? When everything's encrypted was, okay, where's your backups?
Like, do we have backups of these? Do we have anything like that? Well, our server that runs all of our backups was also on these hosts. Okay. So your backup server is encrypted, so we're not gonna be able to get into that. Where was this backup server saving these backups to? Well, that was to another storage device that was also attached to these hosts. So backup drive was a massive VMware file or VMware virtual drive, a VMDK file that this was interesting, Scott. It was
not actually encrypted. It was large enough and the storage was of a size that I don't think there was size to encrypt it, but an attempt was made and it corrupted the entire file. Like, we were able to get it mounted and get back into it, but there was nothing salvageable on the VMDK. Like, folders would disappear as you were clicking around it, and you ran disc checks and there were flags flying everywhere. So, okay. So we don't have backups. What,
what about a Doctor plan? Well, our Doctor plan is when we think there's going to be certain events that could cause a Doctor, we back up said servers to a cloud location. And then after said events, we just delete them because we don't wanna pay to host them up in the cloud full time. So we don't have any Doctor. So what's the plan to get back? And frankly, at this point in time, we had nothing. There's no backups. Everything's encrypted or corrupted. There's no off-site backups.
It's all like, it was essentially we were, do we just rebuild everything from scratch? Like, do we have files? Do we have data? Do we have applications? Do we have all of these where we're rebuilding everything from ground zero? Because there was nothing to restore from. It's kind of brutal all around. Even once you mitigate it, it's, it's not like you can go in and trust your ESX host at that point. You're kind of
paving everything. You're potentially paving your firewalls, It could be paving inline infrastructure, like, managed switches. You you're paving your virtual machine hosts, which have all that other stuff on it, you know, not just the the probably the the the business applications and the surrounding infrastructure, but things, you know, that might be important to a business, say, like your, I don't know, your active directory and things like that. That was that, like, we
have two domain controllers. There was a failover, a primary and a secondary domain controller, but they were both sitting in the same VMware hose. Well, they were sitting in the same VMware environment, same storage, both DCs were encrypted. And like you said, we were finding accounts popping up on firewalls. Like, the breed the attackers had gone in and created, I mean, accounts everywhere, accounts in AD, accounts in switches, firewalls.
And it was like, we found some of these that accounts that actually had like first name, last name, but then usernames are, like looked like named accounts, but we're definitely not employees at the company. So it's a lot of work going in and cleaning it up. And we were able to end up decrypting some of the data, get some of this back, go through and clean it all. But there were a lot of lessons learned, like, even,
we were talking about the ESXi hosts. The password for said ESXi ESXi host was seven characters, and, well, it had a mix of letters and special characters, was absolutely one that if I was going to go try to brute force a ESXi host hosting VMware, would have been on the short list of attempts that would have been made when I was going into brute force it. Yeah. The passwords seven characters. Have you ever looked at some of the password things, like, starting with
this one? Because I think it's interesting to talk through this and some of the best practices and things we talked about as well. Have you looked at some of these for how long it takes to brute force certain passwords? It's very quick, depending on the amount
of confusion that's sitting behind them. So you're often dependent not only on the password and the complexity of said password and all that goodness, but also the, you know, the compute on the other side and the capabilities of a system to mitigate things like maybe, like, password spray attacks
and and bringing that down. And as I recall, like, ESX, one of those things where it's like, oh, I'm gonna have a super sophisticated system for preventing password sprays, you know, compared to maybe, like, the things that you're used to in the world of SaaS products in the cloud. Like, you know, I think, like, Entride, like, m three sixty five Azure are good examples. Like, they have built in passwords, right, protections. Heck, your Google account. Your your Gmail account has it. I would bet
AOL accounts have it at this point. But, you know, ESX on prem, not so much. This is one. So I just pulled it up. Like, number of characters. There's a chart out here, and we can put it in the show notes because this is a chart. This is back from 2021 too. So four years ago, this is from Hive Hive Systems commando dot com website. Again, we'll put that list
out there. Seven character password, if it's numbers only or lowercase letters only, They say four years ago, those are pretty much you can instantly brute force a seven character password. If it's uppercase and lowercase letters, twenty five seconds. If it's numbers, upper and lowercase letters, it's about one minute. And if it is numbers, upper and lowercase letters and symbols, it's about six minutes for a seven character password. Not long at all. To your point, right, AOL, AOL.
You got AOL in my head because my wife still has an AOL email account, and it drives me absolutely up a wall. So does mine. Our wives need we need to have interventions with our wives. But no, like Apple. I'm I know I've done this. My kids have done this, and it infuriates me when they're trying to get into my phone or the computer and they type in the
wrong password. And you get, like, after five or six attempts, and I think they've gotten up to, like, 10 or 12 where it's like you have to wait twenty four hours before you can try your password again. Yes. Every systems should absolutely have that. But to your point, I don't know I have not looked enough at ESXi host lately, but I don't know that they do. And when you look at this chart even, eight characters eight characters is kind of the I would consider eight characters the bare minimum
if it's if it's complex. Eight characters, upper lowercase numbers, symbols, all of that is eight hours. Once you hit the nine, ten, 11 characters, like, eight to nine characters if you're doing complex. So all of those goes from eight hours to three weeks. Nine to 10 goes from three weeks to five years. Ten to 11 goes to four hundred years. Eleven to 12 goes to thirty four thousand
years. Twelve characters has kind of turned into my new minimum, but it still has to be complex because even a 12 character letters only is like three weeks. So a first lesson here, nobody should be doing, in my opinion, less than a 12 character password. And even then, that might be, like, not enough. Most password managers default to something a little bit longer than that. I think, like, if you did, like, a default one password dash lane kinda thing, isn't
it, like, 16 to 20 characters? I know I probably messed with mine in, like, one password, but generally, like, 20 is my min bar. I'm up there too. Mine is sixteen, twenty, 20 four. Like, when I'm creating admin accounts, something like root access to an ESXi host, if I can, if the systems support it, that's the other thing. I've ran into a few times where it's not supported. I'm usually going, like, 24 to 32 characters
and completely random for those. You should never be typing them enough that they need to be short or they need to be easy to remember or they need to be a phrase. Like, you should be able to do a 32 character, 24 character complex password for those systems. Even if the passwords are complex, though, for things like your root user, you still have to worry about your admins and everything else down the chain. So ESX is a little bit of a weird
beast. It's been a while since I've touched it, but you used to be able to do it, like, a bunch of stuff in vCenter. And, you you know, your admins are connecting typically through your vCenter. They're not just, like, SSH ing into the host themselves. You know, you're over on these other privileged systems. Those users might have passwords in active directory. So if your active directory gets compromised because the the these attackers come in, and they're
looking for lateral movement however they can. So they they might come in on the side and all of a sudden go, oh, I'm through your firewall and I'm in your environment. Well, great. If they can land on one of your active directory servers and find a vulnerability there and then they see admins and other things there, they'll just change the admin password for that admin, and they too could have access to vCenter or to SSH in and and come across. So
it's bad once they're inside. Like, it's very hard to stop them once they're entrenched past the edge. Yeah. A %. And another thing that we encountered, I would say another lesson learned because we're on the passwords and credential managers and all of that. Another question that came out was like, where are your credentials stored for all these accounts for not just on prem accounts, but even, like, maybe cloud services or other applications you
have. And they were like, well, we have a password manager installed on one of these VMs that they encrypted. It's doing you a lot of good now, isn't it? Right. One, it's, well, if we can't get back to it, where else can we get some of these credentials? But the other thing is, like, if they exfiltrated these, some of these are not big. You don't know necessarily right away
how long they've been in there. If they're able to get credentials and get to some of these, and if they're able to take that VMDK file and upload it somewhere online and run off with a VMDK file and they have your password manager, they have an unlimited amount of time to spin that up, get into it, and compromise every single credential that was in there. And I think password managers are always one of those I think about. Like, I have my password manager. I'm pretty happy with it. I
think it's everything I've read. It's one of the more secure ones out there. I think we've talked about them before. You've mentioned them already. You can guess it. I don't know that I would ever run an on premises password manager. I don't know. Like, there's risks too of the cloud versus on prem. Right? Like, everybody's always, well, if it's in the cloud, anybody can log into it and anybody can access it. Based on what I've read about some of the cloud ones, they're pretty
secure. The way they do it, the way they encrypt things, where keys are stored, all of that. But your password manager better have complex long credentials for as well. Those better not be a seven character password. You know, the cloud ones are better than there's varying degrees of everything here. You know, I think it's funny going back to how it started. Like, the firewall
attacks are a massive thing. They're always ongoing and going in the background, and it seems to cycle through the vendors as various zero days and things come up. So I thought it was funny when you funny, sad, not funny, You know, when you said, oh my gosh, this thing happened with a customer because I was reading an article back in a couple weeks ago about how Fortinet firewalls were going down all over the place because of zero
days. And I used to work as in a shop where I was installing and configuring Fortinet, you know, and and FortiGates in a in a former life for customers. So I was like, oh. You know, that one was interesting to me because it was old gray hair and gray beard. That was a former life we used to live, but it's definitely a thing that happened back then and and still happens today too. Right? Because there was the one, two it was a couple years ago now, right? SonicWall had a big vulnerability?
Yep. Sonic, Cisco. Every everybody's had their kinda day in the sun when it comes to these types of things. And I think going back to takeaways, the other thing from this was, and we're going this is conversations we're having now with them, is thinking through what are you doing for a backup? Like, you mentioned the whole the one, two, three approach for backups. Like, you need multiple backups in multiple places. On prem attached to your VMs, I would say absolutely
has a place. Right? Like if a VM accidentally gets deleted or you have corruption or self imposed issues on a VM, having them directly attached where you can quickly restore from a snapshot, restore a backup that's local, that's locally attacked storage. Perfect. 100%. But that should not be your only backup as evidenced here. Like, where are those cloud
backups? Where are your immutable backups? That's one thing we've been talking a lot about is why are you not backing stuff up or paying to have backup somewhere where they can't be encrypted because it's a read only copy of that database or it's an or that server. It's a mutable backups of your VMDK files where you can go pull them down in an emergency. Maybe it's not quick because it's not directly attached. Yeah. Maybe you're spending a few thousand dollars a month on it.
Maybe this is costing you 10 or $12,000 a year to keep these up there. And even that is probably, I would say, it's high for some SMBs. It's very low for enterprises. But I would say the cost probably varies in terms of the harm that it can
do. When you look at the amount of time that has been spent on this, the time of man hours, cost of lost business, I would say it probably scales relative to the enterprise where I mean, I spend I probably spend a few hundred dollars a year on backups for myself because the amount of time and effort it would take to recover some of that. Same thing here. It would've cost them probably a few thousand dollars a month, way less than what it has cost in the last two weeks of
man hours to have something like that. That's a good lesson. So so lots of customers sit down, and they look at the cost today. And Yep. I get it's the total, like, oh, you're a backup storage salesperson kind of thing to come in and and and figure those things out. But it's really just it's what's your total cost for that thing, and and what's your ROI?
And like you said, over the course of four years, if you get hacked once and it costs you $50, well, if you only paid, you know, a thousand bucks a year for backups or that, depending on your size, it's well worth it to come through and and get to where they need to be. Yeah. A %. Do you feel overwhelmed by trying to manage your Office three sixty five environment? Are you facing unexpected issues that disrupt your company's productivity?
Intelligink is here to help. Much like you take your car to the mechanic that has specialized knowledge on how to best keep your car running, Intelligink helps you with your Microsoft cloud environment because that's their expertise. Intelligink keeps up with the latest updates in the Microsoft cloud to help keep your business
running smoothly and ahead of the curve. Whether you are a small organization with just a few users up to an organization of several thousand employees, they want to partner with you to implement and administer your Microsoft cloud technology. Visit them at inteliginc.com/podcast. That's intelligink.com/podcast for more information or to schedule a thirty minute call to get started with them today. Remember, Intelligink focuses on the Microsoft cloud so you can focus on your business.
I think it covers that well. Like, 100% have some backup somewhere, and it is absolutely worth the cost. I was talking to another client after it, and it's knock on wood, I hope I don't ever run into it, like, with my home stuff, but more and more of the mindset does not if you have a ransomware incident or a security incident, but when you have it. I feel like they're becoming so common. And it's really one of those things you need to think about is take that mindset of not, oh, this isn't
gonna happen to me. I'm a small business. Again, this was not like a fortune 500 or fortune 1,000 company. It probably they probably fall into the SMB space in terms of size, but it's these attackers aren't out there, I would say, looking for who's gonna be our most profitable person to attack or where can we get the biggest ransom. Chances are they're out there running scripts looking for these zero day vulnerabilities.
And once they get in, they're taking advantage of getting in, not really caring about who it is, what size company it is, any of that. They just wanna get in and get that ransom in. You need to have that mindset of, how am I going to protect myself when this happens? Or how can I best position myself for when it happens to me so that I can be up and recovered quickly so that damage is minimal? All of that. Yeah. It's a rough place to be. So so let's
see. They broke through the firewall. They encrypted all the VMs. They encrypted the backup storage along the way, lost all your DCs and all your other supporting infrastructure. The other thing that stands out to me is the only way to recover from this stuff is to kind of save it when you're done. Because even if you, you know, go ahead and and pay the VIG and, hey, I'm I'm gonna pay and get the decryption key because that's the the thing to
do. And and it seems like that's been happening more and more lately too with, you know, some of them. I've seen some of, like, the bigger hacks on, you know, like, medical facilities and things in The US where, you know, these entire hospital systems are going down and they're just paying it because they have to to get out. But you can't trust it at the end of the day. It's not viable to to, you know, to to go down that path. You're gonna have to come back
in and say, hey. Guess what? It's time to pave that ESX O store set of hosts and and that entire environment. It's time to pave your firewall. It's probably time to pave all your managed switches and and everything else that sits in between. And it could be time to pave your clients as well that we're connecting, Like, even if they weren't encrypted because you never
know what an attacker leaves behind. Yeah. A %. And that's always, like, I mean, I would say one of the first things is once you start getting it back, it's get all your antivirus malware, security detection, all of that up to date to start watching for it. But it is, it's always like, well, where is an account? Where is a random PowerShell script sitting? Where is all these things to your point that you can't really trust and you're always on the alert and there will be stuff continuing on
to try to get it back. The other thing I did not mention this to you earlier, it affected them, but it also would affect somebody like you mentioned a hospital. And I had not thought of this before. I would say because in my scenario, I'm not in it. But as more and more physical devices, you think IoT type stuff, right, is controlled by applications or by servers.
What is the default state of certain physical devices, or what is the situation that happens when physical devices lose connection to the server that controls them? And that was one of the first questions that got asked in this scenario was, this specific stuff, what is gonna happen because the application that it normally talks to is gone? It's, yeah, all these different scenarios that I would say not people don't think
of until they're in them. Again, hopefully from this, people can think through some of this, some lessons learned of different aspects to consider to best protect your organization as you think through some of this? I mean, I think there's a couple ways to one is you can think about it from just the front end of, like, what are best practices and and and what happens along the way.
I think one of the things that always strikes me, like, when I'm having conversations with folks about these types of things, ransomware attacks, you know, business continuity, disaster recovery, they're often thinking about kind of the best practice aspects of it from the front end. I I also like to come in from the back end of things. So if you're gonna go out and look at what are common attack factors for ransomware, yeah, absolutely. Go out and look at that.
You should also go out and look to a certain degree around what does incident response look like? What happens when somebody who's, like, an IR professional comes in, and what do they look at, and what's their checklist, and what do they run through? And a lot of that material is out there on the interwebs as well to go ahead and read through based on your scenario and just how you land and how your
environment composes as a customer. Because your considerations, I think, are very different if you're like that on prem customer, you're maintaining a firewall, or maybe you've got an MSP or somebody maintaining it for you. And then you've got the, you know, the customers who are all in on the cloud and then the hybrid ones in the middle and every permutation in between those. So go out based on your constraints and your environment and and do some of that research.
And then hold up to examples like this as, like, you know, with your finance department and say, it might be worth us spending a little bit of money today to go ahead and not have these problems in the future. I will just say, like, there's a reason why some of, like, the most successful, like, SaaS companies out there tend to be in, like, the BCDR space. Like, if you look at them by revenue, by size, things like this, like, they're not inconsequential
entities. It's because customers, like, they're customers of these Veeams and Commvaults and Rubrics and things like that of the world. The the barracudas, like, as a customer, you still extract a lot of value out of them as well. One other thing I didn't think about, primarily because it didn't apply in this situation, but the other aspect of all of this is to test those backups. Hey. You wanna test them? Come on. Right.
We had no backups to restore from. So it was we didn't find ourselves in the situation where we had backups, but we couldn't get a good restore from them. But you do hear about those as well in these types of situations where they do need to go rely on backups. They have some some immutable backups somewhere. They have some offline backup somewhere. They go to
restore them, and they can't restore. Or the backups are or they find that backups aren't being completed successfully, or there's corruption in the backups, or whatever that might be. I would say that's the other aspect to all of this is not just making sure you have all these different backups and you have plans in place for where it's gonna back up, but if it happens and you need to restore, have you tested it? Do you know it worked successfully?
Do you know the policies that or not the policies, but the procedures to follow to properly restore from one of these types of situations? And, yeah, we talked about offline. We haven't talked about cloud at all. Like we were talking before, use Azure backup. Here's my cloud. We'll get cloud in a little last minute. Like Azure backup, ASR, I'm sure there's stuff in AWS as well. It can back up even some of these on prem VMware environments up to the cloud. It's not hard or complex to set up.
It just takes some effort and some planning on your part. I put an article from Veeam in in the chat, and I'll put it in the show notes as well. But they talk about kinda the three, two, one backup rule and and having these multiple copies and things like that. But there's a section in their article that talks about how to think about this in terms of cloud as well, and how how some of that stuff composes and and what the benefits are to you potentially
as a customer. And I say potentially because your situation's different than my situation, which is different from the next person's situation. So there might be things that, like, I'm interested in from a backup perspective. Let me say, like, maybe I'm interested in offline backups. I don't need instant availability and instant read, and I just want the cheapest option possible. So I'm gonna look for some, you know, kind of archival storage that I can go ahead and put that
into. So that could be maybe like Glacier and s three or Deep Glacier. Could be archive in Azure, you know, and it or it could be one of the equivalent data classes over in, like, a Wasabi or a Backblaze, you know, r two, things like that. So that's one thing. Like, I might go with archive storage. You might go with, say, like, cool or cold because you need instant availability and some of those things along the way. Your redundancy needs are gonna be different than
mine. I might be good with just backup to a single region or a single geo. You might wanna have that stuff spread all over the place because it's really important to you along the way. You might be concerned about immutability. Like, you mentioned immutability a bunch of times and worm storage. So just that write once, read many kind of things. So I wanna make sure that once I write it, that's it. No one else can write to it again, but I wanna be able to kinda read it
back. That could be important to you. Couldn't be as important to me. Like, it all depends. But those options do exist, and they're out there. And I think the options for being able to back these things up and kinda having not only robust backup plans, but the ability to test, restore, test your continuity, things like that, those are just, like, more streamlined
than they ever have been before. Like, I'm always super impressed, like, year over year when I sit down and I look at things like Veeam's integrations, convuls, all this stuff. Like, you know, it's always been kinda like next, next, next, but it gets smoothed out and smoothed out more and more. It's, oh, I could take that VMware VM and back it up over there, but I don't have to restore it back to VMware. I can restore it to a different target and have it go to
go to a different place. Or I wanna back up to multiple targets and coordinate that all on different times. So, you know, I want this backup to be in the form of snapshots or versioning here locally, and I want it to be this other thing and this other construct over here. So, like, the tooling has gotten really good too. There's almost no excuse not to do it other than you're scared of paying the licensing. But I don't know. I think it's always
a good lesson. Like, go out and look at what's happening with other folks when these things do happen. And it kinda sucks to say it, but it's it's really a question of if, not, you know, when it happens, if it's gonna happen. Yeah. If not, when yeah. Nothing. Sorry. It's been a week. Yep. I agree. Yeah. And I think the other way to think about that too, like, you talked about the paying for it. It's kinda
like you pay for insurance. Right? Yes. You pay for insurance, not because you wanna use it or because you I would say you derive value from it, but it's when you get sick, when you need it, it's there. Your backups are the same type of thing. Like, look at it as a type of insurance cost. This is an insurance for when something goes wrong, when you
get attacked, when something gets corrupted, etcetera. And I think too many companies, and maybe this is a message from to IT to whoever's writing the checks, is too many companies or if you're a CEO listening to this or an executive, you think of backups as an IT expenditure versus an insurance
expenditure. Like, you don't think twice if you're a business about writing a check for your liability insurance or your business insurance or workman's comp insurance, all those different insurance policies. But then as soon as your insurance policy is my servers, my infrastructure, my data in lieu of paying for backup, you're like, no. IT can't spend that much. We need to cut the IT budget here because of IT can't spend that much. No. Your backups are it's just another type of insurance
that you should have for your business. Absolutely. It absolutely is. And and depending on how you approach it, you might be able to find this in different ways as well. Like, there's maybe you as an individual or an SMB going out and owning this infrastructure and doing it yourself. Maybe you go to an, like, an MSP. And quite often, like, MSPs will have, you know, varying levels of service plans and things
like that, and they might include Yep. Even additional insurance, like, protections in there or things that come back to you that way, or you can look to the SaaS providers. And I think kinda the the value that they bring to some of these environments. But it's incumbent on us as customers, like, understand, like, our use cases, our scenarios, our constraints, and then kinda line everything else up around
that. But, you know, unfortunately, this stuff, like, it's just become a cost of doing business. Whether you you are actually a business, whether you're an individual, you know, you coming to me earlier and saying, hey. Like, let's talk about this ransomware attack. Like, I still cringe. Like, my so, you know, my on prem NAS here, I I have a QNAP NAS at home, and I had a backup service enabled on that NAS that had a zero day attack in it. My NAS got encrypted
last year. It wasn't anything on my fault. It was a vulnerability in the backup app that came in and did it. Like, I wasn't even, like, exposed through my firewall or anything. Like, I didn't have a specific punch through the firewall to get this thing on the Internet, but it had that connectivity from the cloud service to broker connections back to my NAS, and I lost it and had to pave it. It was painful. Yeah. My wife was not happy the day the Plex server disappeared.
Right? Again, another one of those. Not the if, but the when. Yeah. Did you have it all backed up somewhere? No. I did not. Oh, you did not? Rebuilt it all. Lost my entire iTunes library. That one really hurt. Oh, yeah. See, mine's backed up to Azure. I do have my NAS backing up to I don't remember if I'm going to cold or archive storage in Azure. So this is the thing. Right? I I was backing up to I was because I was using their hybrid backup. I had it enabled. I was backing
up to s three. Yep. But the backups in s three weren't encrypted, and I was only keeping one copy. So by the time I caught it was encrypted, it had already run through, and it had just backed up an encrypted copy of it on the on the Delta. And, yeah, the the whole thing was a mess. Learn from my mistakes as well. Yeah. I was just actually gonna look at my Azure and see what my Azure I don't think that could happen. I'd have to look. I guess that's a downside too
of the right ones. Right? Because if you're overwriting your backups and not keeping historical backups. Yeah. You gotta keep a couple out there. So but but, again, that was my constraint. Like, I didn't wanna pay all that money for that stuff because I was like, oh, it's just my Plex server. Like, I'm gonna lose, like, a couple of MP threes, not my entire
library kind of thing. Or I was just thinking, hey, if a disk goes bad and I I don't wanna wait for the disk to rebuild and I gotta go pull something from s three, that'll be easy enough. But it didn't didn't turn out to be the case. More lessons learned. Alright. Well, with that, Scott, I have a meeting that starts in, like, thirty seconds. Alright. Well, I'll let you get to it. Thanks, Ben. And I will let you go. Thank you. Enjoy your weekend, and talk to you later.
If you enjoyed the podcast, go leave us a five star rating in iTunes. It helps to get the word out so more IT pros can learn about Office three sixty five in and Azure. If you have any questions you want us to address on the show, or feedback about the show, feel free to reach out via our website, Twitter, or Facebook. Thanks again for listening, and have a great day.