Hello and welcome to the let's talk. Azure podcast with your host Sam Foote and Anne Armstrong.
If you're new here, we're a pair of Azure and Microsoft 365 focused it security professionals. It's episode 33 of season five. Alan and I had a discussion around Azure VM builder, which is a service that simplifies creation, customization, and management of virtual machine images in Azure. It automates tasks like software installation, configuration, and patching to make sure that you've got consistent deployments. Here's what we covered. What are virtual machine images and how are they used in Azure? What are the challenges with image creation and management? What is the Azure VM builder and how does it help, and how much does the service cost? We've noticed that a large number of you aren't subscribed. If you do enjoy the podcast, please do consider subscribing. It would mean a lot for you to show your support to the show. It's a really great episode. So without further delay, let's jump in. Hey, Alan, how you doing this week?
Hey, Sam. Not doing too bad. How about you? Yeah, good, thank you. What's happened this week? It's ignite announced this week. I believe that's probably the biggest. Registration open. Yep. Registration open for that. In person? In person. It's an extra day this, this year, isn't it? It seems a little bit bigger. Is that fair to say? Yeah, extra day. And it's now in Chicago, so bigger venue. Okay, nice.
So yeah, it's a little, a little bit more from a cost perspective than it was the last couple of years, but, you know, is a, like you said, it's an extra day now and new location.
Yeah, it'd be interesting to see if. I think it's fair to say we've seen a scaling up of that event, haven't we, since post COVID pandemic? I don't mean in terms of what features Microsoft have released. I'm not talking about the product solutions and the product teams. I'm just saying from an actual in person experience, a lot of it's been focused on, yeah. Remote. So yeah, it'll be, let's hope that it sort of scales back up to the size pre pandemic that we saw. It's always good to meet up and see people in person.
Yeah, it's definitely worth going in person. And as well, I think they've merged inspire now into the same event. So I'm guessing they're saving, potentially saving some costs having two events and merge it into one. But then making this event bigger and better and reusing that funding.
Yeah, I think it's quite smart to have both of them together really because it brings the partner ecosystem, it almost gives organisations that are in the space to two reasons to attend, not just from a, you know, Microsoft and product related, you know, viewpoint, but also, you know, merging in the partner ecosystem as well. You know, you're going to have good representation from partners, you'll hopefully have good representation from customers and end users as well. So you know, it sounds like a pretty good match all around. Really?
Yeah, definitely. We have to see if I get chance to attend this year, so we'll see. Excellent. Cool. Okay, so sam, what's this episode?
This week we're covering Azure VM image builder. Yeah. As a service in Azure, which I've used, well I've used in its native form, I'll call it that quite a few times. But Azure itself has wrapped this image building service well into a service, I should say. So. Yeah. It can be really helpful for people to get started with building custom images in Azure. So I thought we should do an episode on it and talk about it.
That's cool. Yeah, I've done some image building in the past life so yeah, see, let's find out about this service then. So shall we start with what a virtual machine image is and what is it useful?
Yeah, so when you create a server in Azure, we create things called virtual machines and these are virtual representations of what you would traditionally have as a server in your office or your data center as an example. So the whole system relies on. Have we ever done an Iaas episode? We must have done an Iaas episode. I don't think we have.
Probably one of those, it's one of those fundamental things like these storage accounts. Anyway, so virtual machines run on in Azure and other cloud providers, they utilize hypervisors so that essentially the cloud service provider can run multiple virtual machines on an actual single physical server that's sitting in a data center somewhere. The cloud is essentially a, a bunch of somebody else's servers that you rent time on essentially. And this virtual machine is given a number of resources. You're giving an allocation of cpu to give you the performance that you require for your workload, an allocation of memory and also an allocation of disk space. Now this disk space or these hard drives are virtualized now, so like they do with the other resources of these servers that they essentially resell to you. They, each virtual machine has, well each server has a number of virtual disks that can be attached to it. And essentially an image is an installation of your operating system at a specific configuration point. So what organizations do or people running applications is you may start off with, well it's likely that you'll start off with a base image from the marketplace. There are ways to build like custom images. I won't really go into that today, but essentially if you want to create a virtual machine, a starting point is to go to the azure marketplace and essentially spin up a machine. Some images have additional licensing costs. So as an example, if you're running say let's say a Linux VM, you won't have any software licensing costs, not unless you've got a paid contract with one of the distribution providers, like I say a red hat, a canonical, et cetera. So you could spin up like a bare bones Ubuntu virtual machine from the marketplace with the standard image. So you're going to get let's say the latest version of Ubuntu up and running on your machine. And then there are other paid and proprietary operating systems, things like Windows Server as an example, other specific appliances as well think like network appliances that you might deploy or other software that you might deploy as well. These, I'll call them like base images, give you a really good starting point because they're generally optimized for running in Azure and it's what you can build on top of essentially. So yeah, you require an image of for your virtual machine to at least get started with it. Essentially how you create that machine image and what you start with is very important. So one of the really important topics especially in cloud computing is your mean time to recovery in a sort of a disaster recovery scenario. So what you essentially want to do is to create what are referred to as sort of golden images which are as close to your application stack as possible essentially. So in an ideal world you would have say a base image that you start from, maybe that's Windows server. Then you'd install your, let's say, let's say you're running an application on it. You would install all the prerequisites for your application. You would install the application itself. You might have a security baseline that you also want to meet. So the configuration of the operating system to make sure it meets your security standards. So let's say you take your base Windows server image, it might take you, let's call it an hour, 2 hours to do all that configuration. Maybe you're doing that configuration manually, you might have tools to help you with that, but that repeatability, let's say you had to, let's say you had to rebuild that, that virtual machine. How long would it take you to get back to a close state? You know, we do have snapshotting and backup, you know, tools and services that we can utilize. But getting back to that state can take a long time. So that's something that we need to think about. Also keeping our golden images, or base images as we might want to call them, up to date can be a real challenge. You know, you could be on Windows server for many years. You know, like Windows server has long support cycles. So what's the latest? 22. Are we still on 22? Have we gone to 24 yet? Let's call it 22. You could be on 22. Windows server 22 for 2019. Still now, today. And Alan, do you know what the default support lifetimes are on Windows server operating systems?
Yeah, not top of the head, but yeah, it is quite low. I mean 2012, two reals only last year went out of support. Okay, so that would have been like what? Eleven? So it's like ten years, isn't it?
Yeah, ten years probably, yeah. Okay, fine. So you could be on the same operating system for ten years and still be supported. But meanwhile your application, your configuration, your prerequisites and dependencies may patch multiple times a month and you want to keep that up to date. So as you're patching your live system that's in production, your base image slowly, or your golden image slowly drifts out of alignment, if that makes sense. So having a process to keep that base image up to date also validating and verifying that base image is still good is like a job in itself. So, so yeah, you have got to think about how that all sort of you, yeah. Intertwines and how you manage these, these images.
Yeah, they are quite difficult sometimes to manage, as you said, previous life was managing not say a server estate but an end user. One year, Windows XP. Windows XP. Seven and eight and ten, I think. And yeah you're right, you had to build those baselines, those gold images to then put onto your laptop so you didn't have to do much post configuration to get them up and running kind of thing. Or you know, you change software that you're using, you know, your av things like that. So you'd have to change the image to kind of, you know, counteract that sort of thing. So. So yeah, it's useful. But yeah, like you said, it is another job in itself. So I guess that kind of comes into sort of the next question I've got, which is really around, you know, what are the challenges or what challenges are there at least around, you know, creating and using virtual machine images?
Yeah, so I think the first one I want to talk about is how you, how manual that process can be. You know, when you're thinking about, I like to align it to your disaster recovery strategy, you know, what's your mean time? Mean time for resolution. You know, you need to think about if there are ways that you could potentially automate that, that process. But if you are going to look at automating that process, how you manage the tooling, how complex that process is, how many manual steps you've got, what about the underlying infrastructure that you need to manage that process as well? You know, because if you say, okay, I'm going to build a, make it really simple, I'm going to have one master PowerShell script which does everything essentially, right, silly example, but I'll use that. I've got one single PowerShell script that I want to run. Once I've installed my operating system, I want to run it and it configures everything essentially. Where do you store the Powershell script on? What infrastructure do you build? Build, actually run that to start building the image. When you go to the capture the image that you want to make for your golden image, how do you prepare the operating system so that it's in a state that it can actually be sort of snapshotted and used as an image? Once you've built your image or you've got your machine, how do you essentially get it into the compute gallery inside of Azure so that you can start using it? So you have to think about how you manage it, how you build it and how you distribute it. Really depends on the size, scale and complexity of what you're hosting. I would say I know a lot of people now, they will run very lightweight virtual machines and keep their sort of golden image as close as possible. Because in, in the cloud, the lifecycle of a virtual machine can be very different from, from a traditional server. Now that we've got things like virtual machine scale sets which increase the amount of services that you have dynamically, you need to be in a position where you can have your base image or your snapshot for your virtual machines as close as possible, if not at the same level as the other production instances that you've got in place. So, you know, with sort of great flexibility becomes these other challenges that you do need to. Yeah, sort of start thinking about.
Yeah, I think another one in there is consistency. Whilst you might have some scripts, sometimes when you got those manual steps, you may miss them and not realize and correct your image, try to use it and realize it's missing and having to then restart again.
Yeah, and if you need to do that at quite a pressured time, that can also just layer on more sort of stress and. Yeah, and I don't want to paint like a really negative picture, but you know, some infrastructure that we do run is really important to our end users, the businesses that we work for and we serve, you know. So reducing the amount of stress and potential for human mistake I think is a challenge probably worth solving.
Yeah, 100%. And that's that in effect. I know it's slightly different, but when I was building it for, when I was building images for end user devices, we used to create the gold image about 5678 times and we only used to change it every six months. If in fact we changed it even, you know, tried to build it more, you know, more frequent because we had to do it quick or there's just steps that we occasionally missed out of it and you can't just update it, you've got a rebuild. So, yeah. Okay, so tell us about azure virtual machine image builder.
Okay, so these challenges have been, I would say they've been around for a long time and I think with the sort of explosion of growth of cloud and how dynamic cloud is, you know, solutions had to essentially be built to help ops it ops in order to meet the needs that they've got. So I want to talk about Hashicorp, Packer. Hashicorp make a lot of, I call it infrastructure related software. Probably their most popular is terraform, but they also have a, an image creation tool called Packer. Now Packer is cross cloud. It's got lots of different integrations into different cloud providers. And essentially what it aimed to do is it allowed you to have a sort of a third party standard for building these images, these processes, but it would integrate with your cloud provider. So if you wanted say a Windows server virtual machine in Azure, you would create your configuration in Packer and you would say, hey, I want a DS V two as whatever in Azure, give it essentially define your configuration for your machine and then it would use a virtual machine in Azure on your behalf using your credentials. So it spin up a machine, it would do the configuration and it would image it and put it in the compute gallery in Azure ready for you to use. So we had this third party come in, build this relatively popular tool and it also allowed people to say, stay as cloud agnostic as possible because you could have the same configurations and images built in multiple cloud providers. So I'm not going to go into Packer in too much depth, but essentially, yeah, people knew these challenges were there and then they built solutions for it. Now Azure virtual machine image builder is actually built on top of Hashicorp Packer because Packer is open source. So but one of the sort of downsides to Packer is, and I believe Hashicorp do this as well as a paid service, is it's not a managed service. You still need to write the configuration on a machine, you still need to run Packer on a machine as well, and you have to manage the configuration files yourselves, yourself, things like secrets, environment variables, that type of thing is all managed by you as well. So in comes Azure VM Image builder and essentially Microsoft has wrapped a managed service around Packer. So you can bring your configuration settings, your predefined security baseline and software, and you can build what's called an imaging pipeline on top of VM builder and it can all be managed within Azure. It can still be managed via CLI and API, so that you can do things like a DevOps pipeline for automated releases. But essentially you can configure the builder in Azure and you can manage all your configuration there and do automatic builds which it'll automate as well. So it removes the need for any sort of local complex tooling. No, well very minimal or no manual steps. And it takes away all of that complexity and makes it as generic as possible for that definition. If you do have more advanced use cases, you do have the ability to override core image builder predefined configuration. So yeah, I would challenge people to go and have a look at it and see if they can move their current building into it. So there is also a first party plugin as a DevOps task. I believe it's still in preview, but essentially you can trigger an image rebuild from DevOps. What that allows you to do is like do say a nightly build. So what you could do is you could say, let's say you had a, let's say your master image was an Ubuntu image, and every night you rebuilt your master image. You updated all of the dependencies and packages on that Ubuntu machine. You created a nightly or pre prod image and then you push that into your pre production environment to validate it. And you did that every night so you could catch early QA issues as early as you possibly could. Because this system is essentially just building and overwriting images every night. There is cost that we need to talk about with that. But essentially you can fully automate that build so you don't have to wait three months before you patch up your, your golden image and then see if you've got any incompatible dependencies. It allows you to host sort of the customizations and the configuration in various different sources so that you can pull that as part of your build. So your configuration scripts and files that you want to run, they're stored in different containers inside of Azure and it's fully integrated with the compute gallery so you can push images in automatically once they're built. One of the things you have to do is before you create a, if you're building an image from scratch, you have to generalize an image, it's called, which is essentially you take an installed machine and you wipe certain characteristics from it like ids and things like that. It handles all of that for you. So you say your base image that you want to work from, you pick your, your configurations that you want to run and then you do your build and it will automate it for you as well. So yeah, so we've talked about continuously patching and rebuilding your source images. Another great one is that the artifacts that you use for customization are kept private. So it uses an azure managed identity to fetch the resources from your private repositories. So you don't need like a public GitHub link for a script. You can pull those artifacts securely and it's all contained within side of your subscription when that's done. So it's all within your tenancy and your control. Essentially you can also connect VM image builder to your existing virtual networks so that you can communicate with existing state configuration servers such as DSC chef puppet file shares and routable servers and services inside your organization. So you can pull artifacts from sort of your resources should you wish. Yeah, it's available in all of the most popular regions. If your region isn't supported you can build in a different region and then distribute images to outside of the regions. So I would assume as long as you've got, you know, I'll give you an example. It is in all of west us, but let's say it wasn't in, you preferred west us three and it wasn't there. You could build in west us two and you could distribute to west us three when you actually use the images. It also supports all, it's designed to work with all Azure marketplace base operating system images. So it's not just windows, it's not just Linux, and also all of the configurations on top of that. So if you were to use say Ubuntu or windows, you're completely covered there. And yeah, it's just, there's full RBAC in and around it so that you can control who can do what and what it can access. So yeah, it's fully managed for you essentially. Wow.
Okay, I was thinking, so cut. You've kind of given an example of using it for sort of I guess application servers, things like that. The other one I'm thinking of is Azure Vetch desktop Avdae, building those images because you should probably keep those up to date as well. That's a slightly different stance on it because that's more of a user sort of based service and that's where you know, some, that's where image gold images are also key as well. So I think that'd be really useful for that as well, building that every couple of months and stuff.
Yeah, in the documentation, in like the second paragraph of the documentation, it specifically calls out azure virtual desktops essentially because I think how important those base images are and the number of dependencies you can have there can be hard to manage without doing it automatically. Yeah, definitely. Okay, so could you walk us through a, an example of building an image using the azure VM image builder?
Okay. Yeah. So what's really cool is that you can actually do it inside the, inside the portal now as well. In a lot of the documentation it will show you examples of creating it with templates, using those templates from say git repositories to base them from. But essentially what you do is you define your source image, your base image that you want to work from. You then define a number of customizations. So that can be things such as scripts that you want to run, that can also be, that can also be different actions that you want to perform as well. So it could be like install this script, then reboot the machine, then install this script, then reboot the machine and then image if that makes sense. So you can chain together and when you define your customizations, image builder works through them in order that you've defined them in essentially. So yeah, different application installs, configuration, different reboots and actions that you need to perform there. That's me giving a very high level overview. But you can go into insane detail there because you have the power of scripting on those machines. So if you are using, if you do have a set of scripts that you use to configure your machines or at least bootstrap them, it's likely that you're going to be able to essentially bring those across and utilize them as is because essentially all the image builder is doing is creating a target instance and configuring it for you automatically. You can also set any virtual networks that you want to connect to and also you can assign your user assigned identity which has access to all of the resources that you need to interact with. When you've defined that you, you then submit that and image builder then takes care of the rest. It takes your configuration files and it spins up a machine that's equal to the size that you want to. So yeah, whatever family of virtual machine you want to incise, it will spin that up. It'll run your configuration and what you want to run and then it will essentially image the machine. There is also an optional optimization step which you can run which then optimizes once you've done all those changes, it optimizes boot times of the images that can take some time to run. So it is an optional step, but it can sort of pay dividends if you're quickly spinning these machines up as you're using them as an example. Once you've done that, it will then take those images and it'll push them into the compute gallery for you. And your base compliant images are there ready to be maybe tested or, or pushed into production. So yeah, and there's multiple ways that you can interact with the service. You can use Azure Powershell to configure the builder to run it. You can use Azure Cli, you can use arm templates and bicep, or you can use the DevOps task to run these as well. So yeah, if you wanted to run this on a continuous cycle you might set up say a DevOps task or an automation account. I don't know if you can do it with the logic app, you could do it with a function app, I suppose, and run these tasks to rebuild these images. And if you ran it from DevOps, you could store all your configuration items in, in, in your repository itself. So yeah, really helpful. You get the underlining power of Packer, but you also get it wrapped up into a nice managed service for you.
Yeah, okay, that sounds, yeah, it definitely sounds like a benefit if you're building images for sure. And like you said, thanks for the sort of example of how you'd go through it. Okay. I guess the sort of question that we always love to ask near the end is how much does it cost the service itself?
There is no cost for running the VM image builder service itself. You just pay for the underlying resources. So as your machine is being built, you are billed every minute, I suppose, for the machine that's doing the configuration at that time. And applying some of these configurations can take a long time. You might install something which should take some time, you might then reboot X, Y and Z. You might do this optimization which takes some time as well, realigning the disk. So you pay for that as well. And then you also pay egress to submit the images into the compute gallery. And I believe you pay storage for the configuration items that you use. But they could just be scripts, they could also be executables and installers I suppose, as well. And you also pay for storage of. Yeah, the compute gallery items that you create as well. So it's kind of free I suppose is the best way to represent it. You just pay for what you create and what you use.
Yeah, I guess that you'd already pay for the market gallery storage for your images if you built them manually, wouldn't you so agree? Yes.
As you said, to be able to build the image itself, you'd have to have a virtual machine turned on to, to create the image. So, but this is just like you said, automated to the max to, to create it in the most efficient way to reduce those costs as well. Because potentially you could leave your image, your building image on, couldn't you, and leave it over overnight because you've end the day, so you can save costs that way as well.
Yeah, yeah. And you can, you know, if you wanted to, I don't know, let's say you didn't want to recreate your base image every day, maybe your team divide it like every week, let's say. Right, or every month. Right. You can have that run on a schedule like on a Sunday night, and then it's ready for Monday. You're not waiting around for the build to happen, you just get on with UAT and verifying that it still functions in the correct way, if that makes sense. You know, in some teams, I bet you they just reimage their machines and then they don't even, you know, because if you, if you're running your own custom application on these machines as part of your build process, you could take a copy of your like compiled application, push it onto the machine, and as long as you don't have any local storage on that machine, that machine can be swapped out at any time, can't it? Because all that's on that machine is config and application and dependencies and the operating system. So continuously rebuild that and just switch out your virtual machines as and when you need them, essentially. So, yeah, there's definitely really good ways that you can approach that.
Cool. Okay. Is there anything else, Sam, that you want to talk about around VM Azure, in VM image builder?
No. And if you're multi cloud, you might just want to look at utilizing Packer again. There's going to be more for you to manage there. I believe there are sort of paid versions of management on Packer side, but I've never used them. So if you do want to run it yourself, but you want the benefit of the automation and you want it to be multi cloud, then definitely check out Packer, which is the underlying technology. Cool. Cool. Alan, what's the next episode that we're going to talk about?
Yes, I've got two in my mind, so I haven't decided yet. Well, I'm gonna make the decision right now. I'm gonna decide to do, to look into or discuss around how you can put some preventions in place around Microsoft entra, around token replay attacks. We're starting to see a bit more now. We did some testing, what, a couple of, well, probably a month ago now, actually. They're probably thinking about it, trying to work out, you know, what preventions we can put in place. So, yeah, I think it's probably worth having that discussion around how it's, how it's done or a mechanism of how it's done and sort of what the user kind of sees and how difficult it might be to even realize it, but then to also help prevent the, the actual replay of the token. So I think it'd be a good episode.
Yeah. Yeah, definitely. Anybody that manages entra, well, any office 365, et cetera. Right. Any service really, I suppose, yeah, it's gonna be very valuable. Yeah, we're seeing a lot of that in the wild now, aren't we? So, yeah. Great episode that'll be.
Yeah. Okay. So did you enjoy this episode? If so, please do consider leaving us a review on Apple, Spotify, or YouTube. This really helps us reach out to more people like yourselves. If you do have any specific feedback suggestions, we do have a link in our show notes to get in contact with us, or you can comment on our YouTube podcast videos. Yeah. And if you made it this far, thanks very much for listening and we'll catch you on the next one. Yep, thanks. All.