The Better Business Analysis Institute presence, the Better Business Analysis Podcast with Kingsman Walsh. Hi everybody and welcome back to the bit of business Analysis podcast. And today we talk about don't release on a Friday. We're going to be talking about the crowd strike problem that caused wide outages across the world to critical infrastructure which affected Windows systems.
So I'm going to explain what happens and what was the cause of that, what the resolution was, and some thoughts on what we might be able to learn from this experience going forward. So in effect, the recent crowd strike issue involved a faulty update to their Falcon sensor. Okay, so the Falcon sensor is a cloud controlled sensor which is installed on Windows devices, but it's managed to the cloud and this is what caused the widespread disruption on Windows systems.
The issue was identified as a defect and a single content update for this Falcon sensor and this is Crowdstrike endpoint detection and response. It's called which the acronym is EDR software and it's actually focuses on in simple terms, modern age, you know, modern age virus attacks, OK. And this piece of software is very, it's actually quite a good piece of software which focuses on looking for these types of
attacks. This is not like your general run of the more antivirus software, just to be clear. So it, it, it has to have quite important system kind of access to the underlying running of your Windows system. So the software itself looks in there for viruses effectively and notifies you. But in order to do that, it needs access deep within the Windows system in order to do
its job properly. Now, what that opens up is that opens up to a massive risk here that if the software itself is faulty, then you're going to have a problem. So what actually happens or what what the impact of that faulty defect in that update for that piece of software, What it does it did is it caused Windows computers to crash, displaying what we call the blue screen of death, and it prevented them from just rebooting properly. OK, So when your computer fails,
this used to happen. We used to talk about it often with Windows. If you end up with this blue screen, it means basically Windows is saying something's wrong, I'm broken, I can't continue, right? And yes, you've got your own laptop or your own computer, but this is actually, remember that a cloud computer is exactly the same, right? So these could be both a physical computer or a cloud computer.
And a lot of the software and applications and infrastructure that the world's run on is running on Windows servers. So this affected them as well. OK. Or mainly them because it was rolled out automatically via the cloud. And what that meant was that this blue screen, so they've failed if you like, the computer just shed itself, if you like, into this blue screen of death. And then when it tried to reboot, it actually ended up in a loop. It ended up in a loop of boot
booting. It was just stuck there. And of course with a blue screen of death, never got to Windows again. So you could use your mouse and keyboard and it just rebooted itself again and again without really understanding low level IT to be able to stop going to safe mode, which doesn't load all the windows up. It just loads critical factors up. You know, go into the BIOS and knowing how to do that, which people don't have to do that anymore, right.
You know, only if you're an IT professional would you need to do that. So you to fix it straight away, you actually had to have some knowledge of it to fix this. This is why it just took so long for there to be a resolution.
So straight away Crowdsource, you know, told people how to fix it. But I looked at the instructions at the time and those instructions as well, even though they're easy to read, they would not be simple for any standard person to be able to do without IT knowledge and even first line support. Some of the IT people in your company wouldn't know what to do. It would involve them going to someone like who had some IT
administration knowledge. And remember, of course, there's, you know, only a limited amount of people do who do that around around the world. And sometimes that's outsourced to smaller companies, sorry to bigger companies who do that. So you, you know, you have no control over your infrastructure. You had to wait effectively. Now in the meantime, they rolled out some some fixes to fix this problem, identified what the problem was.
Both Windows and Microsoft identified what the problem was. And so they're working on code to make sure that this update wouldn't affect their codes. So writing some kind of catching statements in there to stop Windows from being so affected by it and also crowdsourced in luck to roll out an update. And they had to get all that stuff updated, fixed, tested and then rolled out, of course, to all these servers across the
world. And then once so servers were up and running, then people were checking things like, they're like, were we hacked? Banks were worried about, you know, this was this used as an opportunity? Would have been a perfect time to kind of attack some of these organizations, which is ironic because the software is there to protect them from that stuff. If you're technical. The defect was traced specifically to a file within the update which caused the crash.
And the file was identified as a file called C-0000 two 9/1 Astros dot sys. So there's a there was a specific file that was corrupted. OK, so it wasn't a huge, you know, it wasn't anything major. And this is what happens, right? And and and this can a small change or a small defecting code can affect things. And the resolution was that the engineering team isolated the issue. They put it fixed. So another file to replace that file. That's all really and immediate relief.
They'd how to like what I said, go into safe mode. Windows recovery follows, navigate to drivers, delete the problem to drivers and drivers. To be clear, drivers are software that interact with hardware on your computer. If you've ever heard the word drivers, you need to update your drivers. And of course that's where the problem was. So it's quite low level in terms of code, the code, especially these days with low code. And so that's why it had such an impact to the Windows machines.
Crowd Strike confirmed it wasn't a security incident or a cyber attack, which people thought it was and they were working closely with affected customers. And the CEO of Crowds Crowd Strike has talked about it. He's one of the highest paid CE OS in the world. I don't think he's going to be around much longer, to be honest. And what I get onto just really quickly seeing this is just a update is the fact that what do we, what can we learn?
What can we learn about this right now so that we don't ever do this again? Well, we talk about something in software, in the world of software, if you work in that space and we, we call it don't release on a Friday, never release on a Friday, or there's different variations. And what that means is there's no one limited support in the weekend to fix things. If you roll something out on a Friday and something goes wrong, there's just not many people
around. They've gone out of the office, they're not there to fix. There's no time to resolve something if something critical happens. And what would really that's what's happened in this situation. And we'll find out more as I'm going to make a production is that this is around really bad software development and release. OK, So the fact that there's a couple of things here, one, that crowds Strike could roll this out and hadn't done appropriate
testing. At the end of the day, they hadn't done appropriate testing on enough different devices or enough devices in the same conditions in which were running critical Windows Server updates. They just hadn't done that because if they had, they would have called the buck. So that means they hadn't done appropriate testing.
And equally, it was the fact that the software in which and the hardware in which Crowdstrike is allowed to operate on Windows servers seems a bit extreme that a third party, that a third party which is Crowdstrike, nothing to do with Microsoft, just to be clear, their software, their mistake was able to cause such an impact to Windows service, which is supposed to also be safe, right?
And that can that can open up a whole other kind of worms that this is just one piece of software running on Windows servers, but there's actually thousands of them. So Windows Microsoft need to really look at what power are they allowing or divesting to these third parties and their
software needs to be better. So if they do identify software that's bad, that could impact can catastrophic impact to their product to Windows Server, then they need to write code which is looking for this and doesn't allow an update in this situation. Or it doesn't at least if there is an update and it doesn't work properly, it doesn't cause the
effect that we've just seen. So this is a hard look at software engineering one O 1. It's, it's a good opportunity for us to just look at our own code and our own practices and software deployment. And it's really ironic because we talked about that Y2K last week with business rules and now business rules are so important. Well, this is exactly, this is a business rule, right? This is a business rule in terms of not allowing third parties to release code that affects our
software. I mean that should be a number one business rule. And so therefore there should have been system rules in place to stop that from happening. And so Microsoft needs to look at their business rules from a business model point of view because they are going to be affected from here. They could lose a huge amount of
business. So people go to Amazon or go to Google as a result, and people just gonna be really worried about this and they're going to really have to explain to some of the largest clients in the world why this happened. Crowdstrike is probably the end of Crowdstrike in terms of a company or a trust. And I imagine the CEO will have to step down. Anyway, that's my kind of update, critical update, just in round Crowd strike. I hope you learned what happened and why that affected everything
so much. And it's pretty amazing in 2024 that, you know, our biggest IT outage ever was caused by someone writing a bad piece of code in one file which affected everything, as opposed to a cyber attack or anything like that. OK, guys. See you later.