Hello and welcome to the let's talk. Azure podcast with your hosts, Sam Foote and Alan Armstrong.
If you're new here, we're a pair of Azure and Microsoft 365 focused it security professionals. It's episode eleven of season five. Alan and I had a discussion around Azure Key vault. It is a cloud service that securely stores and manages sensitive information such as keys, certificates and secrets with features for access control, encryption and auditing. Here are a few things that we what is Azure key Vault? What are the common use cases for key vault? What does it integrate with and how much does it cost? We've noticed that a large number of you aren't subscribed. If you do enjoy our podcast, please do consider subscribing. It would mean a lot for us for you to show support to the show. It's a really great episode, so let's jump in. Hey, Alan, how are you doing this week?
Hey, Sam. Not doing too bad. How are you doing? Yeah, good, thank you. Good, thank you. Are you over your jet lag and buzz and excitement of the mvp summit? Over the jet lag? Probably not over the buzz, to be fair. Lots to digest from that week, I think. Yeah, exactly. Lots of info, lots to prep for and things like that. So, yeah, yeah.
I think we got what, two net new products, didn't we, that week, you know, or getting close to ga. So it's more, more to learn, more to integrate, more to use. So not complaining, but yeah, it's always when new things appear plus a bunch of updates. There's just, yeah, a lot to absorb. Yeah. It keeps us on our toes and that's, that's where we. I like to be. Yeah, yeah, exactly. May feel the pain of it, but it's enjoyable. So, yeah. Cool. Okay, Sam, what we talking about this week?
Yeah, so I'm going to cover Azure key vault. Yes. I'm surprised we haven't actually done one on key vault. It's one of those things that, I don't know, I just find it's, it's not that it's not that shiny and sparkly, but it's like a sort of trusted sort of service that we utilize all the time. Right. It's quite, quite sort of central, isn't it? It's quite key to most. To most things as we. Well, in the way we use it anyway. Yeah, yeah, exactly. Yeah, definitely.
Okay then, so let's start with the basics then. So what, so what is Azure key vault?
Yeah, okay. Yeah. So it's a fully managed and hosted service within side of Azure and it's designed to store certificates, keys and secrets. So certificates could be web server, TL's certificates, other certificates that are used in. I don't know what's a good example of other certificates, Alan? End user like device certificates, what else is there? User certificates. There's many different types and formats of certificates that can be stored within it. You've also got keys which are effectively sort of a cryptographic keys, so it supports software protected and HSM protected keys. And HSM refers to hardware security module keys that's effectively where you have a highly available HSM that manages those keys. And I believe it's actually a physical device. Is that right, Alan? That it's actually an appliance that's actually connected in for you?
Yeah, yeah, definitely managed, yeah, managed by Microsoft in the background.
Yeah. If you don't use that, you effectively use vaults which can also, I think also be connected into HSM. But think about vaults as sort of software protected from Microsoft key storage. And there's a whole different raft of different formats of keys that you can store within it. And then the last parts is secrets, which can effectively be any type of secret string value that you might want to use. Maybe you've got a shared secret or some sort of access credential that you want to store within, within key vault. So it abstracts away you having to host anything. It's sort of fully managed. As we talked about, there's an RBAC model with key vault, so you can really granularly define who can access the different types of sensitive information and the level of access that they have. I would say it's quite granular in its approach. As far as Azure resources go, lots of different types of roles that you can set up there. Also auditing and event logging to understand who's accessing what at what time. And because you've got that RBAC model, you also have the auditing and sort of traceability of who has access, not just when they have accessed those types of things. So really it's an Azure resource that you deploy. And keep me honest, Alan, can you have one per subscription? And region, is that the hosting for key vault? If I remember rightly, it's one. No, one per region. You can have multiples. Where is it that the limit of key vaults? I'm sure there was a. You can't have multiple key vaults in can't remember. Maybe that's not right. Apologies if that's completely wrong. I thought there was a limit for key vaults. There is for Azure disk encryption. I'm just thinking that integration, just remember having that bumping into that issue.
That was that it had to be in the same region there, wasn't it?
Correct as the thingy. Sorry, inverse. Apologies for that. So yeah, so it deploys as an Azure resource. So you can deploy them with terraform bicep arm templates. You access them via the Azure portal. There's not a separate sort of location for them and they sit alongside the other resources within resource groups. It integrates with lots of various different services and really dependent on what you're storing, you're getting sort of information specific sort of functionality. So let's just talk about certificates to start off with so you can be a certificate owner. It's effectively a management of your X 509 certificates. You can create secure storage for those certificates and create policies that allow key vault to manage the lifecycle of a certificate. It's probably a big part of a certificate's lifecycle is expiration. We see a lot of, shall we call it, challenges with certificate expiration and renewal. So it allows you to provide contact information for notification of lifecycle events for those certificates. You can assign attributes into those certificates and tags as well to add to the metadata that's stored for them. When you're storing those certificates, you can define key properties, the key types, the key lengths, whether they're exportable, supported key RSA, RSA, HSM, EC, EC, HSM and.
OCT.
If that's, well, that covers the vast majority of certificate key types and there are secret properties that you can put there. When we're talking about those lifetime actions that we've previously talked about, there's a couple of different actions you can do. You can either email contacts and provide contact information, as I've spoken about, or you can set it to automatically renew and regenerate as well. So, yeah, so there's a lot of, there's a lot of really good value there. I've used the example of secure communication or authentication certificates thinking things like intranet, Internet websites, protecting through TL's certificates there. Think about also other IoT and networking devices for authentication and communication as well. So obviously certificates play a vital role in many different areas of sort of enterprise it and also application hosting and development. So having a managed central location that you can access it from is I would say, really powerful.
Yeah, yeah, yeah. Sorry. You use key vault quite a lot. Is there anything on the certificate side that's um. You don't think I've covered there? No, I don't think I've not to be fair. I've not reused the, the certificate side of it. I've more been the secrets side of it in my usage. Yeah, yeah, they're gone.
I was just going to go back to um, kind of like this. This kind of sounds very, you know, a very important store of sensitive information in some form, isn't it? And you kind of talked about RBAC being quite granular. Is there any protections around like an own, you know, someone getting an owner or a contributor role and then being able to see these secrets? Or is it, is it quite locked down in that sense?
Yeah, the, the RBAC model is completely disconnected from. Yeah. Like the owner role, contributor role, etcetera. With inside of Azure and dependent also on the policies that you've got in place as well. You can restrict owners not having RBAC access to grant permissions. Isn't there an actual role for the permissions admin for key vault? I think that's how they do it, isn't it? You have to have a specified role in order to manage key vault.
I think you can put, I think Anona can put the permissions onto it, but they can't see the secrets without correct giving themselves that role and things they don't get by default kind of thing. Yeah, wasn't it?
Yeah, exactly. So yeah, there is definitely another level to that. And yeah, I suppose that's where your arbach in Azure comes in, isn't it? Right. You know, just do you have sort of bad hygiene of, of RBAC in Azure? You know, are you, are you segregating out those duties and roles correctly? Because for me, a lot of the systems that need to interact with these certificates, keys and secrets are not humans. Right. Maybe if you manually need to renew some of these, you know, values. But let's, let's say you've got a secret, which is. And I'll go into, I'll go into, let's talk about secrets. Right. And then I'll revert back to my example. So a secret is just effectively a string of up to 255 characters. No, sorry. That's for the content type. So you can set a secret and set its content type for when you're reading from it. And what you can do is you can set a number of attributes to it. So you can do an expiration date on a secret. You can also have a not before date as well that can be passed in. So that can give you a hint to say the secret data should not be retrieved before that date. That is only informational though that field. So you have to respect that with your integration. And there's also an enabled attribute as well. A couple of other read only attributes are created and updated, but not too much else there. On that side of the things, there's also secret access control. So there's a full set of roles and permissions for actually reading, writing, listing, deleting, etcetera, those secrets. But as we were mentioning before, a lot of the time, a lot of the time, access to these, these things. Let's say you had a secret, let's say you had some non rotating shared secret that you had between two systems, effectively a password of some passphrase of some sort, and it never expired. Then it might be that the person that created it in the first place, you know, seeded it with the original secret information. But then maybe all of the reads after that point should be like a managed identity of some sort, you know. So when we're talking about auditing and logging and also RBAC, we can really granularly define who should have access to what and when they should have access to it. Like an example, when we create key vaults via terraform and, you know, the actual keyvault itself is created, the secrets can be pre populated by sort of the identity that runs that terraform, that terraform build. And we can assign like only a managed identity a role in order to access that. So in theory, if you, if you don't have something that needs to be manually updated by a human, then in theory you can completely lock humans out of needing access to those secrets. Right? So it's, I would say really well integrated from an access perspective.
Yeah, definitely. Yeah. I do love, do love key fault. Yeah. And it's probably worth just sort of calling out as well, is we sometimes use secrets to store other values, don't we? Not just secrets, because it is a good way. I mean there is a new, for Azure, for app service, there's a new, what was it, hosted configuration manager or something like that. I think I covered it in my app service. Or was it a news update? News update.
There's a new update for config, a new service for config which looks really good, but you can also mix and match. A secret doesn't have to actually be a secret value. It can also be used as like a key value sort of configuration store. Right. So it's pretty powerful from that perspective. I don't know what scale you can get to. I did have a note of the maximum length of. Let's have a quick look. Maximum size of 25,000 bytes was 25,000 bytes. 25. Oh, sorry, wrong keyboard. Apologies. 25,000 bytes in megabytes. Hardly anything at all. So I'm not sure what the actual maximum length will be. I suppose it depends what type of characters you're placing within it and the content type you've used. Maybe. Yeah.
There doesn't appear to be a limit on how many secrets, keys or certificates you have either, from the look of it. Maybe on the, on the mashed HSM there is for obvious reasons of capacity on the hardware, but maybe that should. Be a script to see test the limits of it. Cool. Yeah.
Okay, so you know, like I said, like I was sort of saying before, this seems really powerful to store these secrets in us in a secure way. I guess. I guess the question might be is how would you know a developer, an organization, you know, a service, you know, store, you know, this sensitive information without key vault if, you know, if they're in azure, because I'm assuming there's the equivalent in the other, the other clouds.
Yeah. So I think we'll call it sensitive. It's probably not a good idea. This is sensitive information, right. You know, keys, secrets and certificates. It's obviously on prem solutions for sort of like certificate management, but from the sort of application development world that I've lived in, secret management and key management has been tricky to say the least. Do you find a paid solution that you self host and secure? Hashicorp have got vault, which is like a, I believe it's still open source. They were having some changes with their licensing, but that's like a self hosted like key vault sort of equivalent. But I think the challenge is when you, when you run things yourself you obviously get the, you completely manage it yourself, right. So you've got to think about the risk both ways. You know, you are giving this sensitive information like for instance the, you know, the TL's certificate for your website as an example. It's fully managed, you know, and hosted by Microsoft. Stored and managed by Microsoft. You know, you could approach that conversation in a way of, well I don't want them to have access to everything, the complete keys of the kingdom, so to speak, like literally. So there is that sort of argument. But I think the other side of it for me is you're either not managing these very well, you're storing secrets in config files, environment variables, you're manually copying certificates onto web servers, keys. Right. You're pasting keys into portals and I don't know, backing them up somewhere. So would you run your own solution and is your solution highly available because, you know, if a web server doesn't have access to, to a certificate, then it can't, can't communicate correctly with, with clients. It might be cached, but the point is, is that, you know, do you have a highly available solution and also do you have the resources to run it and also secure it as well? So there's definitely pros and cons each way. I think we can all agree that as long as you're moving towards some sort of system to manage this type of sensitive information, then that ultimately is the sort of best way to move. Yeah.
Okay, cool. Okay, so you kind of talked about this earlier and you know, we've got, we've got a key vault, we've got the secrets in, you know, how or what does it integrate with, you know, how easy is it to, you know, to call the, call the secrets and the certificates etc.
Okay, so to start off with there's a bunch of client libraries that you can use. So there's a.net library, Python, Java Spring and node js out of the box access. So you know, if you're using one of those, if you're using one of those systems, then you're good to go. There's also a rest API as well, which you can management dot Azure dot azure.com and you can get access in to list all the various different actions that you need. So kind of. However, if you are building bespoke or integrating, you're really well covered there. Let's talk about other integrations. So if you're using Azure DevOps, you can reference key vault secrets in Azure pipelines. So when you're defining a pipeline, you can be tempted to either store secrets sensitive values in the actual pipeline configuration itself, or you can use what are they called in pipelines? Are they called properties or can't remember what they're called? Configuration items, variables in, let's call them variables, they might have a different name, initial pipelines, which those values aren't stored in the actual config itself, they're stored in Azure DevOps. They can be marked as sensitive on the runs, but the people that have RBAC access to the pipeline itself still do have access. So you can make a service, I believe it's a service connection in Azure DevOps to reference a key vault and pull its values in. It's also in a very similar way integrated into Azure app service and Azure functions, which allows you to effectively bind in a key vault to provide secrets and configuration into, directly into the app service application itself. So that's really, really powerful. That's for secrets. There's also the same for certificates in app service. As you can imagine, hosting web applications in app service is really important, you know, for, for public facing and also non public facing APIs and web web applications. Azure Kubernetes service has an integration for key vaults which allows you to inject secrets and certificates and keys into Azure Kubernetes service. Not something I've personally used before, but that integration is there. Same for Azure databricks allows access to secret management directly into key vault. And spring integration allows you to read secrets from Azure key vault in a spring boot application. And also in spring you can also secure spring boot apps with Azure key vault certificates. Spring is not a system that I've ever used before, but there is an integration there as well. That's more, I would say developer orientated integrations. But again, first party PaaS solutions on Azure. Azure key vault also integrates with virtual machines. The example that I used previously was for Azure disk encryption. So the encryption keys are stored within key vault and used for Azure disk encryption. And also it is, and I didn't know this, but it's also integrated with managing storage account keys. But apparently that is a legacy integration. So I'm not sure what the rotation out of that is and what the new system is for that. I've personally never used key vaults in that scenario.
Okay. I was just thinking about some of the other integrations because you've got also got, I don't, I think you mentioned it, we've got things like logic apps you can use it with as well to pull keys into that, to then do API calls if you need to. Or does it? No, it doesn't work the other way and manage identity to access it. I think the other one that I know about is probably API manager where you can use named instances and reference a key vault that way as well. But yeah, those integrations at least from the portal side, obviously not from a coding side. And within Azure, like you said, sam, about function apps, things like that are relatively easy, aren't they, to get it hooked up. It's all just making sure you're using the manager identity to access it. And it's relatively simple to get that, at least with the function app anyway, that you create the configuration item or line, isn't it? And then a reference to the secret. And then you can use it in your code, can't you?
In effect, yeah, exactly. It's probably worth calling out networking at this point as well because I think that's probably the only slight stumbling block I think is probably fair. Is that fair to say? So it, you can, can you bind it to a VNeT, the firewall or is it always publicly rooted? I know you can do private link with it but if you have link. Then it can hook up to a VNet then.
Yeah. And I'm not sure if it's a. But basically you know, the main struggle really it's not the portal, it's more the connectivity I would say. You know how you are. It does have a firewall functionality that you can configure. So that is the only thing. If you are, if you are locking down who and how you can access it, you sometimes can trip into edge cases like is it fair to say Alan, some services which can't at different tiers can't Vnet. You may have issues with private transit into key vault and vice versa. You may have to transit externally but not externally if that makes sense.
Yeah, you might have to. Yeah there are some services like well function apps and logic apps in there like consumption tiers. You can't hook them up to a Vnet. So yeah you'd have to transact over the Microsoft top Internet layer, the public layer there. Maybe not, you know, exiting their services themselves but within azure. Yeah you're right but you know you can't. I guess the other sort of answer around that is that you can't access it without having permission there either. So it's not like. Yeah, yeah it's not like you can't, you don't, you know, it's, it's an ominous sort of access sort of thing. You know, someone can't try and poke it. You can't accidentally do that. It is by default you need to have permission to access that key and provide you know, an authentication mechanism for it. So you have got that. Yes. In some, in other scenarios you may have to lock it down to vnets and things like that. But generally when you're, some of the SaaS services just needs to be accessible from that point. And securing your identity is key as well, isn't it?
Yeah exactly. Cool.
And I guess as well because I guess probably a scenario around it being sort of centralized and that is that if you are storing like for a certificate example, if you're using the same certificate maybe it's a, maybe it's a wildcard or something that. And you can store that in there you're using for multiple services then you know when you go time for that renewal and you have to do it manually. You know, at least you potentially only have to do it in one location and not have to go to every service thing with the secrets. If you've got a client id and secret from entra, you know, at least you have to change it once if you're keeping the same secret across the service itself for all the different components.
Yeah, yeah, exactly. Yeah. So having one source of truth that your other, that your other solutions sort of call into. Yeah. Is a great way to manage updates and distribution. Okay, I guess we kind of talked about some of this, but can you give us a working example of using it?
Yeah, let's think about an actual, so we've talked about sort of web applications using certificates pulled from key vault, sensitive configuration items, things like connection strings, database secrets, that type of thing bound into app service and function apps. I'm just thinking about logic apps, not specifically just for logic apps, but storing application ids, client secrets for entra applications and spns in your logic apps as an example. We do use that quite heavily, passing around credentials. I think, I think any scenario where you do need to store those types of, those types of sensitive information is pretty well supported. I do really like the fact that you can offload Azure data encryption into key vault, which means that you've got access, you've got access to auditing and sort of more centralized management of Azure disk encryption keys inside of Azure. So that's another really powerful scenario I think as well. There wouldn't, there won't be anything stopping you using key vault in a different location or even potentially cloud service and calling back because you can, you can peer it, you can quote publicly, access it, it's not public because there's a firewall there and authentication. So it really is a flexible solution there for many different scenarios.
Yeah, I guess in theory, like you're saying you can pull it from anywhere, provide, you can authenticate kind of what we were talking about before. I guess that means that potentially you could store those secrets for services that might be hosted on premise in data centers kind of thing as a central store for those secrets to secure them. Kind of like what you're saying you could potentially build services on premise, but actually can you just use that cloud service there that's been tried and tested?
Yeah, yeah, exactly. Okay, so probably the question then is because it sounds, you know, it sounds like an obvious thing to have if you're managing sensitive sort of secrets and things like that within an application sort of environment. So I guess the question is how much does it cost?
Okay, so there are two different tiers. There is standard and premium. So let me just see if I can get a breakdown of the differences between the two of those, because I'm just. Because the pricing page has got. It is exactly the same. Do you know the difference between the two, Allen? Is it just like hsms? I think it's just HSM is the difference?
Yeah, I think so, yeah. Okay, fine. So, yes, secret operations, right? I've got to get these numbers right. Three cents. And this is in dollars transactions for secret operations. So. Yeah. Will you use that? I've personally never, I've never really seen, especially if you were caching some of this information, like if you were using it for a config for a web server or something like that, you might cache it into memory so that you don't need to keep coming back to key vault anyway. So don't know certificates, if you want to be able to renew certificates, renewals, or $3 per renewal request. Any other operation on certificates is the same. Three cents per transactions. Three cents per ten thousand transactions. Sorry. For software protected keys. So RSA, other advanced RSA is three cents per ten thousand transactions. More advanced key types above 2048 bit encryption bit keys. So RSA 3072 and RSA 4096 and ECC keys, elliptic curve cryptography keys. That's fifteen cents per ten thousand transactions.
Doesn't sound like it's breaking the bank still at some point.
Well, yeah, I've never seen a high. All the scenarios I've ever done have been quite low transaction volume. Still valuable systems, business systems, but they're not pulling hundreds of thousand transactions per day in key vault. So I'm not really sure about the high end of. At what point does it make sense to build your own, you know, but if it's like three cents per ten thousand transactions, you got to be doing some serious transactions to even get to like a virtual machine hosting costs to run your own, you know, open source solution or container. Yeah, definitely HSM. So HSM protected keys, one. So for RSA 2048 bit keys, it's $1 per key per month as a base cost, and then three cents per ten thousand transactions. Then for the advanced key types, you know, the 3072, 4096 and ecclesiastical, you're looking at the first 250 keys is $5 per key per month, and then the transactions again. And then as you go up, it scales up. So between 251 to 1500 keys, it drops $2.50, 1501 to 4000 keys, drops to keys. And above is forty cents per key per month. But you got to buy a lot of keys at higher rates before you get to there. So I'm guessing that's quite, quite expensive, like the certificate renewals. If you want to do key rotation, it's $1 per schedule rotation. And I believe in order to do a HSM protect, you want to do your own dedicated pool of HSM because I think HSM can be like Microsoft shared and managed. You could also do managed HSM pools, which are like actual, I think they're the dedicated hardware that's $3 20 /hour so.
Yeah. Okay. If you've got, if you've got the regulatory. $3.20 times 732, it's about 2000, $302,400 a month, basically. But I think you've really got to have a regulatory compliance reason to go down that route. I would assume. You'Re probably going to talk about it in a minute maybe. But if you've got that pool, does that mean that you don't pay per certificate, you just got paper transaction or something? Because you're in effect paying for the.
Yeah, I don't know. It's not detailed. Let me just see if I can grab a quick look at the docs because there's a whole section for it here. Just give me 2 seconds. Because it kind of, you kind of understand the per key cost. If you're sharing it, you're sharing the, the cost of the HSM pool. But. Then again, it's still quite a lot. You'd have to use, have a lot of keys to make it the same unless you've got the regulatory compliance to be separated. Yeah. I don't know is the honest answer. Okay.
It's not in the faqs either. I've never got to that level, so I can't, can't comment on it. But my gut is it's all compounded on top of each other, I reckon. Okay. But that might be wrong.
Okay, cool. So it doesn't sound too, too expensive to at least work to get started. It seems something to nothing to sort of test it out and things like that and start building applications. And then it slowly ramps up depending on what part of the key vault you're using and depends how much calling, especially with the secrets, how much, how many times you're calling the, the key vault to get that, to get the various secrets in there. Maybe that might start ramping up once if you've got a large service and providing to a large customer a large amount of customers.
Yeah. And it's probably worth calling out that this is just a system for storing this sensitive information, you know, you have to generate by, you know, these certificates and keys yourself. It's really a distribution and access control mechanism and storage mechanism for them. Yeah. Okay, cool. Is there anything else then, Sam?
No, that's it. Because it's quite a relatively simple service, really. Because it only stores, what, three different things? It's just integrated with a lot of stuff. Right. I mean, some of those concepts are complex. Defining certificates, keys, rotating them, managing them. There are some complexities there, but from an actual application perspective, storing data and making it available to other systems. So, no, that's it. That's it from me.
Okay, cool. Just probably just one thing around it is that it has got things like purge protection and soft delete, hasn't it? So that you do delete them. Things like that. At least you can get back, is not to say caught us out, but when we've been developing and we set these services on and we want to nuke it and rebuild it with the same name. Because it has to have a unique name, doesn't it? That was probably the thing you think about before. Yes. Has to be a unique name worldwide, doesn't it?
Yes. True. Yeah, but, yeah, we've created them, deleted them, gone. Okay, I need that name again. And we can't use it for 30 days.
Yeah, exactly. And it's a lot of recommendations, isn't it, in like defender for cloud and policy to enable purge protection. Right. Because it's like good standards to make sure that you can't nuke, you know, your keys and your certificates. Right. Just in case. And secrets. Just in case. That's your central store for them. Right. So, yeah, and like you say, so, yeah. Pro tip on your actual development environments. Make sure you add that as an exception until you go into probably staging and production because. Yeah, it has got us out a few times.
Yep. There's always a one or a two on the end. Yeah, exactly. Yeah. Key vault one, ABC and then D. Cool. Okay, so what's our next episode? Yeah, so next episode is our news episode. We do that once per month. So next week will be into April. So we'll recap on the Azure March news. And I'm sure there's probably a lot to cover.
Yeah, yeah. It's definitely been a few announcements about a few things, so that's good. Okay. So did you enjoy this episode? If so, please do consider leaving us a review on Apple Spotify. This really helps us reach more people like yourselves. If you do have any feedback or suggestions around episodes we have a link in our show notes to get in contact with us. Yeah. And if you've made it this far, thanks ever so much for listening, and we'll catch you on the next one. Yeah, thanks all. Bye.