Hello and welcome to let's Talk as. Your podcast with your host Sam Foote and Alan Armstrong. If you're new here, we're a pair of Azure and Microsoft 365 focused IT security professionals.
It's episode 41 of season five. Sam and I had a recent discussion around patch management for servers, an everyday task for system admins that can be simplistic and complicated at the same time. Here are a few things we covered. Why is patch management important and what are some of the issues you can run into? What tooling can help with patch management for servers? What is Azure Update Manager and what is hot patch for Windows servers?
We've noticed that a large number you aren't subscribed. If you do enjoy our podcast, please do consider subscribing. It would mean a lot to us for you to show your support to the show. It's a really great episode, so let's dive in. Hey Alan, how are you doing this week? Hey Sam. Not doing too bad. How are you? Yeah, good, thank you. We're in December now. I assume you're in full festive spirit. Something like that, maybe. Don't think I've had the time to be fair.
Is the tree up? Are the decorations out? Is it like a grotto in your front room? Where are you on the Christmas theming?
I suppose the tree, the tree is up, the reef is on the door. But the. That's not my doing. If it was me. It doesn't get put on till the 24th because I forget normally. I think this time my parents have come around and purposely made sure it gets up there. So I should get some, some usage. Yeah. Light decorations. I think it's probably fair to say because we're, you know, planning to move in the new year. So I think we've, we've been more relaxed around it.
I was getting links, I, I was getting links for the most ridiculous sort of Christmas decorations we have resisted buying. Like I saw like a, there was this, there was this five and a half foot tall like Nutcracker basically. It was like a life size and it was just too expensive. I was like, no, that's ridiculous. It would look great in the hall. But like it was just, it was ridiculous. So no, we haven't, we haven't splashed out at all this, this year. We have been quite, quite reserved which is, which is good. Anything else in from the Microsoft Micro sphere? Microsoft Sphere. That's new this week, Alan?
Not that I'm aware of. There's been a few previews sort of Starting out again as you know, as they continue to develop stuff in the background from the CCP program. But no, I don't think there has been too much coming out from what I can remember anyway. Have you seen anything?
No, no, not particularly. I, I just assumed we'd have a bit of slumber after the whole ignite because it, you know, it's so close to Christmas. Well, you know, the holiday season, you know that I just assumed we wouldn't get much, to be totally honest with you. Not that the machine top stopped spinning or you know, churning, but I think they've done quite a few like our last. It was our last episode, wasn't it? The ignite and update episode. So yeah, you know, there was, there's a fair few things in that book of news, wasn't there? So. And just generally in the month as well. So no, not, not, not that much excitement on the Microsoft front for me. What, what are we talking about this week?
So we're going to talk a little bit about patch management, but mainly around Azure Update Manager and Hot Patch. It's something that, it's quite somewhat interesting I think for people who have to sort of manage patching of servers and things like that. As we know, sometimes it's a, it's a full time sort of job.
Yeah, I kind of feel like servers in some respects are, I don't know, quite a few years behind maybe the, I don't know, consolidated integrated approach that Endpoints now have, if that makes sense. I don't know whether it's just the complexity of like server environments, if that makes sense. But it can be tricky, can't it, with different, especially across different operating systems and different platforms that you're hosting these machines on.
Yeah, yeah, definitely. It's, it's different to end users for sure. Yeah, yeah, yeah, 100%. Right, yeah. So should we get started? Yeah. Okay. Could you just sort of start us off then with sort of Patch Management 101, you know, what is it and why is it important?
Yeah, so it's kind of as it sounds. It's patch management is the management of patches. It's a terrible way of sort of describing it, use the words in a different order but in effect, you know, within, within an organization, be it, you know, user endpoints, software, things like that. Servers know, there's always, you know, these Windows updates, there's always patches to go out, you know, improvements to, you know, the operating system or the software or probably the more critical one is, you know, security updates when vulnerabilities are found or there's. Yeah, there's this, you know, potentially bug fixing within some of the applications. So for an organization this is very critical and I kind of, it's probably split into sort of two or three areas. I think there's probably sort of software patching, so third party software patching and then you've probably got operating systems patching which then kind of splits into two around I would say user endpoints and then servers as source separation. So it's important to sort of do this, you know, to do patching and keep up to date. One is to make sure that you're in, you get all, like I said, you get all the latest features or updates, you know, to updates to the software. But like I said, the main one is to make sure that any vulnerabilities are then closed and you know, the software is protected until you know, maybe the next vulnerability is found within that, that version of it. But from a day to day sort of, you know, business as usual kind of thing, you know, it is a sort of a full time job for a security team, you know, sec admin for the servers or for user endpoints to sort of manage to make sure everything is being patched and it's not breaking any of the services that might be on it. Now it's very fairly difficult to, without any tooling to help you sort of visualize what you know, what's patched, what's not patched, you know, the order that you're going to do it in. Because these are all the things you kind of got to think about when doing patches is the scheduling because you might not want to patch. Say you've got a cluster of web services, you may not want to patch all of them at the same time in case that patch breaks your, yeah, your cluster of front ends, you might only patch one first to make sure that you know, the application still works. At least then you can roll back or at least rebuild that one machine and not worry about the others kind of thing. So there's all these things you've got to think about when doing patch, when patching. And I think that's kind of the difference between user endpoints and servers is that potentially if you patch one machine for a user endpoint, you only affect one user, that user might not be able to work for that day because it's crashed, etc. You're only impacting an individual user where for servers, if you patch, like I said, a web cluster of web servers that could take down that service that you've got as your product, but then that could be servicing thousands, millions of users, you know, across the world kind of thing or you know, internally and then that service is gone. So the criticality of the patching becomes, you know, it's very, you know, you have to be very cautious about how you sort of do that patch management side of things.
Yeah. So I suppose there's a, there's two sides of it, isn't there? There's this patching for sort of vulnerabilities and management of that, but also to get new features as well. So I suppose there's, there's multiple reasons to, you know, have a mature patch management process. Right. I suppose. And I think to your earlier point that, that there's just too much software now to do that manually. The scale is just too enormous, you know.
Yeah. And it's, it's not like it's every three months, every six months. You have to do it, you know, for Windows updates, for, you know, for Windows servers, that's every four weeks, once a month that you've got to sort of run through that process. So.
Yeah, and I suppose, yeah, and I suppose the criticality of these endpoints, because they're servers, they, they, it's not like a user endpoint where if you ask them to reboot at the end of their working day to install like a Windows update, that's not really the end of the world, is it? But servers are typically on 24 7, aren't they? Serving workloads. Yeah. And serving multiple users as well. Yeah, yeah, exactly.
Yeah. Okay. So yeah, could you just sort of, you know, talk us through the types of tooling that's out there to help, you know, manage patching on service.
Yeah, so there are, there's probably quite a few server management tooling out there that are third party to sort of Microsoft tooling. I don't know what some of those are called. I think you might have like potentially like patch my PC might be able to do servers. Seen that more on the user endpoints. As an example, you might have something like I think a Ninja one, sort of a device management, server management kind of tooling. But from a Microsoft perspective, probably the most common one that we see is Config Manager in its various sort of naming used to be called SCCM System Center Configuration Manager. And I think it was called Microsoft Endpoint Configuration Manager. And I don't actually know what it's called which, which terminology it's using now, but we just call it Config Manager in effect. So that's got a mechanism in there for doing patch management and then I guess you've also got the mechanism of sort of probably Windows, Windows Update Server, WSUS Server, which actually is going as being decommissioned I think. Is it next year or the year after? I think that service which everyone, you know, that's been the sort of primary way of doing patch management and managing which patches go to which servers kind of thing. So that's probably been the more traditional, I say traditional, but the more traditional on premise sort of way that you would you manage updates on there outside of trying your best to sort of, you know, manage it yourself? It depends how big your estate is really depends on how you know what sort of tooling you might do. But another one is that kind of come out in the last couple of years, it's called Azure Update Manager as another one that potentially can be used for Azure and on premise as well.
Yes. Do you want to take us through Azure Update Manager and the benefits that it brings?
Yeah, sure. Okay. So it went generally available in September 2023, so it is probably G8. It's just over a year old from a GA perspective. So as it sounds, it's an Azure service that's in the portal. The idea of it is to help manage the updates for all machines and to be fair, this includes Windows and Linux for patching. But the good thing about this is that it's not just for Azure, it's also for on premise and other cloud platforms. So it's using Azure ARC to cover those other areas, so other clouds, things like that. The idea is it's a single pane of glass sort of concept for your patch management. You can do remote checks which affect assessments against your endpoints to be able to see whether they've got the latest patching on them. I think you do that, it does it automatically every 24 hours, but you can just do it instantly. You can in effect build your dashboarding to show sort of the information you need to see. It's kind of RBAC based. So depending on whether you have access to the servers in Azure where they're ARC or Azure based shows you that information. So it's quite easy to highlight who the responsibility side of things. There's very little from an onboarding perspective because in Azure it's pretty much sort of enabled anyway in some form. And arc, as long as you've got Azure ARC deployed, then it can just be, you know, enabled on that, on that environment. There was, there was a previous version where you could do patch management in Azure using automation accounts. That's not needed anymore. That, that sort of service been deprecated as far as I'm aware. So now this is the new service around it. I've talked about roles and things like that. The idea as well is that if you've got extended support for some operating systems, this is done through this update manager as well. And yeah, I think it's just generally that it's a single pane of glass across the whole lot. I think it's quite interesting. It also does Linux as well and in effect you can build your schedules, your sort of patch groups, things like that to help you with. Yeah, that, that sort of, that patch management side of things.
Yeah, nice. Yeah, to, to consolidate it and make it as easy as possible. It feels like, you know, workstation endpoint management, you know, sorry, not endpoint management, but you know, patch management now. Right. If we're just having one singular place and Azure ARC coming into, you know, as a technology that gives you the, let's call it. It's an arc, not a bridge. But a bridge is an arc, I suppose to other clouds and even on prem, you know, at least you can modernize that process and get it cloud connected and cloud managed maybe without ripping out or migrating workloads into Azure if you don't want to.
Yeah, I mean, in effect what it's using is using sort of Windows updates for business from the, from the actual, where it gets the updates from. It's just this is actually managing, you know, when they get them and that side of things. It's kind of, I suppose, kind of manipulating the local, sort of the local group policy, I suppose if you could say that when it patches, things like that. Yeah, exactly. On the fly kind of thing. So. Yeah, and it's one of those things that, you know, you can, you can have that maintenance, you know, that schedule, but if you need to patch it, if there's an opportunity to patch, you know, outside that schedule, I think you're able to then, you know, push it out to then, you know, do it there and then kind of thing if you get that opportunity.
Nice. So I suppose the million dollar question, or hopefully not million dollar question, is what's the licensing like in the cost day? So if it's in Azure, if your machine is in Azure, then it's free, which I think is quite cool. Well, it's very cool because it's free. Is it free though or is it included in the cost?
Well, I suppose you could say it's included in your virtual machine costs. The management plane underneath it, etc. Etc. Yeah. Okay. It's included in your Azure costs in general, you have to pay extra on top. Yep. For Azure ARC enabled servers it is $5 per server per month. So there is a cost there. Okay.
But Azure Update Manager is included with Azure benefits for Azure Stack HCI extended security updates enabled by Azure arc. So if you are doing extended security updates for an older operating system, you get it included. And Microsoft Defender for server plan 2. So I think, okay, I think if from plan 1 to plan 2 on Defender for Server, I think is, I think it's $5 for plan one and $15 for plan two. So it means it's only technically costing you $5 more in cost that you'd need for an ARCS ARC machine to get all the other benefits of Defender for Server Plan two side things.
Yeah, well that's quite a strong, that's quite a strong value add, isn't it? If the standalone is $5 and the uplift is say $10 or even if you just didn't. Yeah, yeah, that's true. Yeah, that's pretty good. Yeah. Yeah. And, and as we are seeing more and more people consolidate down into that singular Microsoft like security ecosystem, it could well be that they've already got Plan two in a lot of places. So you know, for those types of customers, that is just an extra included benefit, you know, on top. Yeah, definitely.
Yeah. The, the Xerox serves are charged by the day, so not hourly, but it's like 16.2 cents per server per day basically. Which then amalgamates it to $5 per server per month. Yeah. If your service turned off, then you don't pay for it.
Okay. Right, yeah. Because virtual machines, they're not like containers, are they, where they've, they can potentially have really short life cycles? I know they probably could because you can still do sort of batch workloads, can't you, with VM scale sets and things like that. But if you would, if your VMs had such a low sort of time to live, then you might not even need an update solution like this. Potentially. True. Right, true.
But I suppose if you sort of jump boxes, maybe you don't turn on all the time on premise. Yeah, I'm just, I'm just thinking it's, it's, it's more of a rare case that you're using a virtual machine for 20 minutes, isn't it? You know. Yeah. Or, or only for a couple of days even, really. Yeah. I don't know. The stats, that's just My gut is that typically virtual machines are left on or available during business hours that, you know, at a minimum.
Yeah, I suppose, I suppose if you had a RDS solution in your local data center, on premise data center, and maybe they're only on nine to five, but then you're paying daily, so maybe they're not turned on at weekends. You can get a couple of days free. Yeah, nice. Yeah, really good. Anything else on licensing and cost? No, there's nothing around that. The only thing I was going to say is that it can also interact with hot patching.
Yes, yeah, did, sorry, did I, did I, did I skip over that? Because we did. We were meant to cover that. No, no, it's the next question. It's the next thing on the list. Oh, okay, cool. Yeah. So yeah, can you expand out on that on Hot patch for Windows Server?
Yeah, so this is, this is quite interesting in that hot patch has been out for, oh, I don't know actually, maybe, maybe it's been like a year, maybe a year and a half. So hot patching in effect is a different way of installing security updates into Windows Server. So as it kind of, as it kind of sounds, maybe these updates can be installed and binaries in the memory can be updated without needing a reboot. What?
Yeah, so I think that there is obviously going to be some patches that need reboots, things like that. It's not getting rid of everything, but from a security patching side of things, it does it whilst the machine is on with no downtime and actually applies the updates as well. It's not just they're applied and requires a reboot later to then actually be active. So this has come out, it was, it is only for Windows Server 2022 and 2025 and as and it has only been in enabled for Azure machines only and it depended on which image you used previously. So beforehand there was actually a SKU that was like the 2022 data center Azure Edition Hot Patch and Hot Patch Small Disk. And originally they were the only two that could do it. At that time when it first sort of came out, that's now changed to in effect, I'll just read these out but 2022 data center Azure Edition Core, Azure Edition Core Small Disk, I said the hot patch ones and now it's including 2025 and basically in effect the same thing. Azure Edition Small Disk Core and Small Disk, so they're now enabled for that. So now in Azure you can apply these hot patches without needing the reboot. I think it kind of entailed that maybe it's every three months, maybe you could get away with now for a reboot outside of normal reboot requirements from other software, things like that, you know, in general rebooting, but not a requirement maybe for, for Windows updates. So that was pretty cool. I think that's going to make, you know, it makes it really easy at least get those secure updates out there quite quickly without disrupting, you know, services, things like that. But what was announced from what I remember at Ignite was that it now allows you to use Azure ARC for Windows Server 2025 data center and standard to do hot patching on premise, which I think is key.
So you think Microsoft is so hybrid and multi cloud focused now? I know some tech you can only do in Azure but I kind of feel like they're just like meh, you want to keep it somewhere else. We'll still give you the functionality to do it if you want to do it, you know and we'll license it to you. You don't have to. Not all roads lead to Azure if you don't want them to, you know.
Yeah, that, yeah, on premise, hybrid, multi cloud kind of feel. Yeah, definitely. I think, I think you're right. I think stuff comes out in Azure first because first party they manage the underlying infrastructure and things like that so it's easy for them to push stuff out. But yeah, you're right. It's almost like the second phase is always oh, by the way we do it using Azure Arc, you know, on the, you know, for your. On prem and multi cloud sort of environments.
Yeah, it's like cool, you can leave them there but we still have solutions for you that you can license. Yeah. And what also came out of Ignite around hot patch was that also I think the next version or the latest version at least Insider you can do hot Patching with Windows 11 on, you know, on prem. Well it is would be on prem but you know, normal end users which I think is amazing.
Great quality of life improvement, isn't it? As an end user myself, I can totally get behind hot patch egg, that's for sure. Yeah, so I think that is on the insider builds at the moment. I've not seen it on my work laptop yet but I probably need to do a next version. But yeah, that's, that's going to be I think interesting on that side. Not maybe not requiring to do a reboot for three months. I mean it probably takes three months for admins to try and get users to reboot. Yeah.
You know, try everything you Know just to make them. But yeah. So difficult. Yeah. So yeah, I like it. Really good. Really good.
But yeah. So like I said, it is, it is Windows updates, patches. It is, it doesn't do. It's not a part of current so corner Microsoft documentation. The following types of patching aren't included in the current. Sorry. Patches currently aren't included in hot patching, which makes, makes it quite interesting. They're going to try and do more sort of other updates. So non security updates for windows.net updates and non windows updates such as drivers, firmware updates and so on. Which that kind of makes sense that you're not going to be able to patch the firmware and drop. Well, maybe drivers, but firmware without a reboot. So. Or maybe on a Surface you could. Because they own the OS or the, you know, the hardware. Who knows?
Yeah. So it's, it's, it's, it's going to be used for security vulnerabilities and patches, isn't it? For core Windows, essentially. Yeah. And, and I don't know how often those hot patches come out, to be fair, because that could be weekly and you wouldn't even know, would you, from that point. Because you wouldn't care. No, you wouldn't care, would you? Could be daily in some form if you really. If it really needs to be.
Yeah, but. Yeah, but how do you want your hot patch today hourly? Well, I'd just like to remind everybody of the crowd strike outage that more recently, sorry, dung on CrowdStrike, but we. And insert any other EDR name that could have caused the same issue, you know, just to be politically correct, but you know, do you want your endpoints patched immediately with hot patches? Play devil's advocate. Don't know. I don't feel like as an organization that I need them immediately. Do we? Don't know. That's probably something to be debated, I suppose.
Yeah. But again. But whatever your stance is on that, and that could be a non issue, but whatever your stance is on that, it's cool technology, isn't it? It is just cool that it is possible to now do that. You know, I mean some people might say that other operating systems can already do this. That. That is true. Yes, that is true. But, but yes, for Windows it's big.
Yeah, I agree. I don't think for me it makes a difference if other operating systems can do it. I think we should celebrate any improvement. Right, because. Oh yeah, absolutely. You can't stones and grass houses, can you? Oh, no, no, it's great. It's definitely great. Yeah. But yeah, I'm sure there's a few, there might be a few listeners that think, yeah, well, you know, Linux, I think Linux can do it anyway sort of thing.
So I, I, I, I was, I, I was thinking that I was going to make a comment but I thought actually you know what, it's, it's not, it's not worth it. I don't want to be that guy, but whatever. Even if it is slightly delayed and should have maybe come years ago, anything that can help, you know, administrators administer more easily is a tick in my book. You know, the fact that they've, they're moving towards it, they're adopting it because I assume the engineering behind that is a lot bigger than we could ever imagine or comprehend. Right.
So yeah, it's, it's probably the, the way that the operating system has been previously engineered, things like that. You know, it's just thinking about how you modulate a lot of it more now, isn't it?
Yeah, yeah. And I don't think, I don't want this to be a Linux versus Windows server like debate, but I don't think you can criticize Microsoft's for Microsoft for their backwards compatibility of their kernel. You know, they, they, they drag forward a lot of technical, I'll call it technical debt of backwards compact compatibility, you know, and they should be commended for that, I think. You know, it doesn't matter which side of the fence you sit on, you can still be appreciative of, you know, the engineering that goes on, you know, of the systems that we use day to day because it doesn't matter what's more popular. But you know, these technologies power, you know, massive enterprise and you know, the world so to speak. You know, so that's, it's a really good addition.
Yeah. So yeah, that's, that's really a thing I was gonna sort of COVID on this one. It's semi short episode, I say that, but I think it's just a bit to sort of bring up because I think we've kind of talked about all the, some of the other areas within Azure from the services side of things. But this is more of a maintenance tool I suppose. Yeah, management tool that's within Azure, so and it is security related in some form because you know, patching is unfortunately something that everyone has to do.
So. Nice. What episode are we having next week? Ellen? Sorry, Unless there's anything else you want to cover on it. No, there's nothing else. Next week's episode is going to be our Yearly finale. So the last episode of this year, episode 40. What was this one? I can't remember. 42. It'll be 40. 42, yeah. Does that mean we've done 42 episodes? We will have done 42 episodes this year. Yeah. 10 weeks off, Alan. That's not good enough. Right? That's mental. That's amazing.
Yeah. We started doing three a week, though, didn't we? So three a week? Oh, three a month, you mean. Sorry, three a month.
Sorry, three a week, Please, Alan, you're a taskmaster. No. So, yeah, season finale, we take a break over the holiday season to recharge. So, yeah, we'll do a recap next week of maybe our favorite episodes, favorite tech from 2024 that we've seen, some of our predictions and maybe plans for 2025. So, yeah, it's no tech, just sort of us rambling for, I don't know, 20, 30 minutes. But yeah, we'd like to do a recap and sort of just reflect on. On each season as we go.
Yeah, I look forward to that one, actually, because we've done a lot this year, I think. Yeah, exactly. Exactly. Cool. So did you enjoy this episode? If so, please do consider consider leaving us a review on Apple, Spotify or YouTube. This really helps us to reach more people like yourselves. If you have asic feedback or suggestions, we have a link in our show notes or you can put a comment on one of the episodes on YouTube.
Yeah. And if you've made it this far, thanks ever so much for listening and we'll catch you next week. Yep, thanks. All.