Welcome to the Azure Security Podcast, where we discuss topics relating to security, privacy, reliability, and compliance on the Microsoft Cloud Platform. Hey everybody, welcome to Episode 66. This week, it's myself, Michael, Sarah, and Mark. Gladly, they've taken a little bit of a break. We have a guest this week, Joey Snow, who's here to talk to us about some identity stuff. Rather than talking about identity in terms of people, we're talking about workload identities.
So with that, let's take a little lap around the news first. Mark, why don't you kick things off? So a couple of interesting things that we released recently. For those of you that are ancient like Michael and myself, you may remember something called the Immutable Laws of Security. In the process of switching over from TechNet to the modern docs now LearnSite, some of that stuff was dropped into an archive and really hard to find.
So as we were in the process of resurrecting it, we brought it back. We also realized that there was this need for a new layer or a new altitude of them around cybersecurity risk, because we saw some really key truths that needed to be understood, that debunk some myths. So we put those up there. So just the AKMS slash security laws, of course the links will be in the show notes. We cover a lot of things like not keeping up is falling behind.
Disrupting attacker ROI is really the goal of security. Don't solve a technology problem, don't solve a people or process problem with a technology solution, because it won't work, things like that. So that's out there. Really love to get some feedback on that and see what you all think. Another thing that we're working on is the opposite. So the opposite of a best practice is anti-patterns. Because sometimes those are really illuminating.
Like, hey, I didn't realize we were doing it wrong, but now that you describe it that way, that does sound dumb. So we've been really collecting some of those. I've got a little LinkedIn post going on, so I'll send that out for the folks that are interested and have some to contribute. Some of my favorites are collection is not detection, and we got a whole slew of them for patching there. Then compliant is not secure.
I can't tell you how many people I've met that think that, hey, we're compliant, so therefore we're secure. Gotcha. Not really. But just all sorts of things. There's also the user world of we're using the same password, all throughout the sock and identity world and all that. So we figured there's a lot of good ones out there, and we'd love to get your thoughts on ideas on that. Then the last one I just wanted to highlight real quick is around the CISO workshop.
So that is something that we've had the videos posted for a couple months now, and we've been training some folks internally, getting them all ready to deliver. So this is something that we actually do deliver, not just as that one recorded time on the internet that you can enjoy in perpetuity. But also we do that delivery with customers, for its CISOs, CIOs, security directors, IT directors.
Oftentimes bringing teams together, helping them connect and build a common strategy, understand the latest trends in Zero Trust and how do you actually apply that? What does it really mean to a program, to a strategy to all those elements? So that one as well as our end-to-end security architecture, the architecture design session one, are actually something that we're starting to do deliveries on. Shadowing and reverse shadowing and leading a couple of those.
So, hey, you might get lucky and get me there, or unlucky as the case might be. But that's really all I got this week. So I guess it's my turn then. So I've just got one bit of news. I love AKS and Kubernetes nearly as much as I love Sentinel. Just a quick one, there's a public preview of being able to rotate your SSH keys in your existing AKS node pools. So of course you should be using SSH. It is the default in AKS now.
You have the default for Linux VMs in Azure, but you can now rotate those keys on existing node pools and you don't have to re-image the node. So that's very good, doing your good security hygiene and best practices, much, much easier. And that's just me for this week. Yeah, I got a few items.
So the first one is, as your front door now supports managed identities, and I've probably brought this topic up on every single episode, it's really great to see more and more PaaS services support managed identity so that way you don't have to manage your credential and accessing other resources. So for example, if you had to access a storage account or a key vault and you can use the managed identity of the front door instance to restrict access to that resource.
And the nice thing is that the actual credential is actually handled by AAD rather than your application. So great to see you as your front door now supporting managed identities, but that is in public preview. Next one in general availability, I did not even know this one was coming. And this is actually really awesome to see. And that is that there's now the ability to view the tables that make up log analytics. Historically, the tables have always been sort of hidden away from you.
You didn't actually realize that you know, certain data was being collected in tables underneath, and then you could then use log analytics to query over those tables. Well, now you can see the tables. You can see what the fields are. You can see basically everything, you know, that the schema that makes up the table. You can even create and delete new tables if you wanted to. I'm not saying that's a great idea, but you can.
So this is great to see you because it's sort of, I don't know about you guys, but I like to know kind of how things work. And when things are sort of hidden away from me like that, I don't know, it just, the penny doesn't drop sometimes. So it's fantastic to see there's no NGA. You can now actually look at the tables that make up your log analytics workspaces. The last one is also NGA, is we've now defaulted to a rule set 2.1 for Azure Web Application Firewall.
So these new rules basically are aligned with the OWASP core rule set 3.3.2. And they also include some other protections that have come from our own internal Microsoft Threats Intelligence team, the Mystic team. So you should be looking at that new rule set. And if it makes sense for you guys, we'd go ahead and turn it on. Okay, so I get to kick this interview off with our guest this time. I'm gonna let him introduce himself, but funnily enough, he is in fact my boss at Microsoft.
So that's why we're all gonna be on our best behavior. But this week we have Joey Snow. Joey, do you want to introduce yourself? Yeah, but I promise I'm not gonna be on my best behavior. I mean, that's just so abnormal to me. So hey, and thanks for having me. I'm Joey Snow. I am a Principal Cloud Advocate Lead.
So basically I lead a team of security advocates, which is NetNew inside the advocacy world where we very similar to what a developer advocate does and our operations advocates do where we go out and advocate on behalf of Microsoft and then back into Microsoft, particularly around our security products. Wow, I had no idea that's what you did, Joey. No, not so ever. I had no idea. But this week you're gonna talk to us about workload identities. So, you know, let's start with a basic question.
Like, what the heck is a workload identity? How's it different from a managed identity? Like, tell us all the things. First, like what is a workload identity? So the best way to think about this is to break up identities into kind of two things. You've got like human identities, right? So those are like your employees, your external users, those user identities. And then you have machine identities.
So things that could be like, you know, a device identity or an IoT, robot process automation is an example. But another piece of a machine identity could be a workload identity. And we think of these things like application to application or service to service calls. Things that run in the middle of the night.
An example of folks who've been around long enough, there's an exchange service account that we created back in the days of doing on-premises exchange or the old school service accounts, the Microsoft service accounts that we have, are an example of a workload identity. Things like your data backup services are using to gain access to the systems to run at night. Your CRM systems typically have them security software. So it's just an example of, you know, applications.
You asked about the managed identity piece. A managed identity can actually be part of a workload identity and it's actually a better way of handling it than we have in the past. Again, yeah, I wonder how I get to ask some of the most basic questions, but here we go. I mean, what, I mean, ultimately, you know, what is the problem statement here for workload identities and what sort of problem are we really trying to solve here?
So really what it comes down to is that non-human identity attacks are on the rise in the world. When you look at things like ransomware, data exfiltration, when you look at leveraging applications to impersonate, you know, a known application that then can become compromised and impersonate things, takeover systems or be used for, you know, lateral movement. But really it comes down to we have spent a lot of time over the last couple of years in really focusing on human identities, right?
We're monitoring human identities, we're enabling MFA and we have those principles around that. What we have not been doing is spending a ton of time on what's happening in the background, in these applications. So we're really only paying attention to the things that we see, it's kind of like this streetlight effect, right? So as we've hardened user identities, well, our attackers are basically going, well, that's hard.
And attackers don't like things that are hard and difficult, at least in most cases. So when we make our user accounts less susceptible to things like phishing and passwords, brain other things, the attackers are shifting their focus to other things. And so now it's like, hey, look, we need to start paying attention to this. There's a lot of bad behaviors that we had done in the past, just go back to the early days of the service account.
You really didn't have to worry about them too much, right? Cause we had our four walls and nothing was exposed to the internet. Now things have changed. So this whole thing about workload identity, so I've been around Windows security for way too long, Sarah keep, don't you say anything? But it's interesting cause you run Windows applications as an example under some kind of identity. You always do.
So for example, it could be a service and you can run a service as like a service account, which is actually like network service, local service, heaven forbid system account. They can also run a specialized user account, right? With specific group memberships and privileges. And that concept has been around for a long, long, long time. Even in the same in Linux, right?
You can run, say for example, the Apache demon, you can run the main HTTPD process as roots because you need to open up port 80. Then you can have other HTTPD demons running as say nobody, right? And they don't need to run elevated. And I've seen this so many times, it's like your opinion on it, where you're reviewing like a design or something and you see a service running as, say as an admin or even a global admin, right?
Like a domain admin as opposed to a local admin or perhaps even a system account, which are highly elevated accounts. And if that process is compromised, then the attacker is now running a system or as root or as global admin or local admin or something, which is, it can be absolutely catastrophic.
So, just like your opinion, and it's just something that you see where people just say, well, we run this account because we run it as domain admin because it's always run as domain admin and everything kind of works. Well, of course everything works because you run as domain admin or a system. The problem is, so does the attackers code, right? The attackers code runs as a system or as domain admin, and it just works because the process that they've compromised is running elevated.
Is that something that you see a lot as well? I mean, is that part and parcel of attacks that go on? Yeah, it very much is. It's very much a common area of concern, right? Where things have been no shock to anyone. Things are over permissioned. That's because the nature of how we've been kind of dealing with security, right?
Security tends to be bolted on and added on at the end instead of becoming part of building an application or building out a service or part of just that natural process where it's born with security in mind and designed in that way a lot of times, things just kind of get tossed over the fence and it's like up to the security folks or the operations folks to then secure it afterwards.
And you go back to applications that were designed in the 90s and the early 2000s that still exist today where things, it's like a user. A user insists that they have to be running as domain admin. That's not the way it should be done anymore. And so yeah, over resource or sorry, over permissioned identities or workload service accounts are absolutely a big problem. A lot of it has to do with managing it. You mentioned that service accounts could be user accounts. That's not a good thing.
There's a reason why Microsoft created back for the on-premises piece, the managed service accounts, where basically there's not a username and password that has to be cycled. If that password got out in the wild, then I don't have my entire infrastructure at risk. And then you start looking at things like if you've got to have certificates or you need to have a username password combination using things like Azure Key Vault to store that, where at least it's in a single place.
And then you can, earlier you mentioned managed identities, where you can leverage those managed identities on top of because with managed identities, you don't have to manage these credentials. In fact, the credentials aren't even available for you to do anything with.
And then it allows you to use that managed identity to authenticate to the resources so long as those managed identities can leverage Azure Active Directory authentication and when we're talking about the Microsoft view of managed identity. So yeah, it's very much about that.
It has a lot to do when we start looking at things that indicate security concerns from a workload identity place, we're looking at things like permissions, we're looking at things like when our change is being made to the application, what type of changes are being made, what are the credential types, and then configuration changes, things like changing URIs or log out URLs modifying or changes to who owns an application and environment, assuming that you actually know who owns the applications
in your environment, which is another key thing that people want to take a look at. You know, it's interesting to bring up the whole thing about users requiring admin access and so on. One of my pet peeves is developers thinking they need admin access just to develop software. Most of the time you just don't, you really just don't. I've heard people say, well, I need to debug my application and Windows debug is a privilege and only admins have debug privilege.
That is true, but the debug privilege in Windows doesn't mean you can debug code. What it means is debugging a process that's not running as you, that is totally different. Like if you was like single stepping through some code that you've written, like a console app or something like that, and you don't need debug privilege whatsoever. Even if you say debugging a service in Windows, which is running at a different identity, you can actually grant your accounts as a user the debug privilege.
So you're still a user, but you have a dangerous privilege. Debugged privilege is a dangerous privilege, but you're at least not an admin. Yeah, I see this all the time. And honestly, I get into a few little, let's just call them discussions with developers and development managers. You know, when I see a lot of developers running his admin, I realize a bit of a soapbox points of view there, but yeah, if you're an admin, if you're a developer, please just don't run as admin.
No, and listen, we've seen the likes of 25 plus million log-in credentials that get stolen through things like malicious applications. So your credentials taken, and you've got these elevated credentials because you insist that you need to have that. That's just an easier way for attackers to do things. And as I mentioned earlier, the easier it is for them to do it, the more they're going to target that environment.
When you put things like MFA and having lower credentials, and when you need to elevate into those other things, doing things in just in time, where it's, you know, there may be a process where you have to, you know, maybe have an MFA challenge in response, but it's time bound. So at the end of that, I don't forget that I need to log out of that elevated process. It's just good security hygiene.
Yeah, and the way I kind of think of that space is like that, you know, infamous Spider-Man quote of with great power comes great responsibility, right? And ultimately, you know, there's this sort of, I mean, I've developed this sort of allergy to permissions. I do not want any privileges or permissions that I don't need. I once had a customer offer, you know, in my couple hundred thousand user environment, do you want enterprise admin rights?
You know, and I'm like, no. So, you know, I mean, and we just have to sort of teach that, I think, to a lot of those developers that, hey, you don't want it to be your account that caused the data breach, the disk, the ransomware, whatever the bad thing was. So like one of the things I wanted to ask you a little bit about more Joey is, you know, some of the non-human identity attacks. If you want a couple of examples, is there some other angles or some other kind of attacks that you've seen?
Yeah, so kind of traditional threats that we see leveraging application identities is, I kind of mentioned it earlier where you've got a compromised application. There's a legitimate application that exists in the environment and it's been hijacked. Another example of application identity threats is where there's an application that's been placed into the environment that is malicious. So it's created by an attacker just to do bad things.
And the reality is, is malicious apps don't belong in any environment, right? They need to be eliminated. And then we need to go back and understand what happened to allow that in here. And then the most popular thing is a misconfigured application, right? So that means that there is some sort of a configuration, whether that reside inside or outside of the identity service or within the application that has made it susceptible to compromise.
And we've seen tons of examples of these types of things. If you look at things like supply chain compromises, right? Where something gets placed into a legitimate application and then it hides everything else that they're doing, including adding in a malicious application. You look at things like consent phishing, which is a technique that tricks a user into granting a third party application access to their account.
And they can either impersonate the user directly or they can use that victim account to phish other people. They can use things like some directories and some configurations allow anyone to consent to an application being added into the directory service. That's bad behavior, we need to not do that. We really need to manage who can actually add applications into things like Azure Active Directory.
And then there's kind of the, it still baffles my mind but you have a look at developers are still putting credentials into code. And then they're taking that code and they're checking it into public code repositories. Yeah, but the good news is, I mean, there is tooling around, right? That can actually find some of these things. I mean, it's great to see tools like GitHub automatically scanning repos, looking for that kind of problem, but people shouldn't be doing it in the first place.
So Joey, do we see a delineation between how people manage their user accounts and how they manage workload accounts? And then I suppose ultimately, do we have some kind of tooling to help with this? And so first and foremost, you don't need to, so do we have tooling? Yes, absolutely.
When you talk about some of the stuff that has, we talked about it at Ignite around, workload identities and workload identity protection, but the reality is, is regardless of the tooling that you have, there's some simple things that people can do. So the first thing is understand one, what are the application identities in your environment and to who owns that? So when things go south, you know who you can call in and partner with.
I was doing a presentation down in Houston and chatting with a user group and doing this talk on workload identities. And I asked who understood, you know, the apps in their environment. I think I saw three people in the back of the room raise their hands. And afterward, people were coming up to me going, this is one of the first things I'm going to be doing when I get back to the office.
Because again, all of that attention has been paid on user identities and nobody's been looking at what exists in this space. Is this, because this is a, you know, a ripe attack surface if you're running say a service from, you know, 2001 that's running as a domain admin, nobody knew. It's got elevated credentials. The password probably exists inside of some document that's sitting on a server.
So if, you know, somebody gains access to that, there's the data and there's another way that you can continue to elevate up and move across things. So, you know, it's understanding what exists there. And then really applications, they're pretty easy to kind of track. Their activities, you know, they typically exist in a specific location, whether it be an IP address or a particular country or particular region. And they typically have very predictable patterns. They run at certain times.
So they're very easy to keep tabs on. So it's not one of these, once you identify them, it's a matter of, okay, hey, these application identities, we can move to more of a managed identity where we're not thinking about it where, you know, secrets are being rolled and we don't have to pay attention to it. But maybe there's some older apps that exist in the environment.
So instead of trying to solve for everything, move what you can to managed identities or, you know, using things like KeyVault or, you know, those type of services. And then if you've got some of these other things, take a look at what you can gain from things like sign-in logs and audit logs, where you're only looking at, say, yeah, we've got these three apps. We've got to keep tabs on them. All right, great.
Well, we know that it runs on this server and this data center in this region and that it runs between these hours. Like a backup service that runs, you know, typically predictable hours. And then just keep your eye out for anything that's unexpected, right? The credential type has changed. Wait, wait, we used to use this, you know, certificate based auth and now it's moved to a user password. That's like a hello red alert.
What is that resource accessing and should it be accessing that resource? They're really predictable. So you can see anomalies very, very easily. And, you know, look at, oh, hey, look, this application is all of a sudden trying to grant itself higher permissions or higher privileges or an end user is granting an application consent to do something. These are all things we can kind of take a look at.
And when changes are made to application credentials, are they made outside of business hours or outside of the normal process? Who made the change? Who was the owner of that? Again, it comes back to knowing these things. And then finally, when we start looking at bad behavior like credentials and code and so on and so forth, you mentioned there's tons of scanning tools in it. Some of it is built in. There's open source available ones. There's on-prem versions. There's cloud ones.
They're customizable. You can build your own or you can use some of the more scalable vendor-owned tools to kind of do some of that work. Yeah, you know, I really want to stress something. You know, you mentioned key vault, managed identities, and I really want to stress this because I think it's just so important. So managed identities for clients' authentication are absolutely the way to go because AAD handles credentials. You don't have to. There's nothing you need to store somewhere.
And honestly, even if you're storing it in key vault, key vault just turns a crypto problem into an authorization problem. That's all it does. So how do you access the secret in key vault? Right, you got to authenticate it to key vault to be able to access the secret. So then, you know, key vault can apply its RBAC policy to determine whether you have access or not. So if you're running a process that needs to access key vault, then you need to run that process if you can as a managed identity.
The good news is, you know, more and more pass services are even IaaS. I mean, VMs can run as a managed identity, but you really, you know, if you're architecting something on Azure, there are my two rules of thumb. Server authentication is always TLS. Anything but TLS is varying degrees of wrongness from TLS. I mean, I'll give you SSH as long as it's done correctly. And then for clients' authentication, it's managed identities.
I'm not really too concerned about whether it's a system managed identity or a user managed identity, but the correct answer is managed identities because if an attacker gets in the environment, there is no credential. The credential is not there. It is completely managed by AAD, which is over there somewhere, that the attacker has no line of sight on. So I really want to stress that. It's just critically important.
It really comes down to treating your workload identities like you do a user account. Use really good security principles. Try to use those strong credentials, as you were mentioning. If you've got to use certificates, use X509 certs, and make sure that these credentials have short lifespans, and that they're not committed into code repositories, and they're stored securely, you have to do credential rotation. It's good health. Use least privilege.
Don't be running as admins if you don't need to run it. Don't give a service right access if it just needs to read something. And then have that, as I mentioned, this life cycle management. Use just in time elevation, type techniques, and then monitor. Look at these applications. If you've got apps that have poor hygiene, and they need to run with highly privileged permissions, keep tabs on those. Have a look at what is actually happening, and then go back, figure out what you have.
Take a look at what exists out there. There's probably some stale apps and credentials that are old, that have just been existing. Get those cleaned up. Make sure that the access that these accounts are using is the access that they need, so recertify them. And then, like I said before, if you can't lower those application permissions, like from ReadWriteAll to ReadAll, then monitor those things.
Okay, I want to add something because you said something which I totally agree with, but I want to add some more. If you're communicating with a process that requires an extra five-nine certificate, and obviously the private key, and that's all it does, then you better make sure that that private key is locked down in some thing that uses a managed identity to access that particular credential.
I really can't stress that enough because if you're using an extra five-nine certificate with private key, with obviously with a private key, you've got to store that private key somewhere, and you need to make sure that the account that needs that is using the correct authentication, and then obviously as you are back on top of that, that authentication mechanism better be AAD, which is using a managed identity.
I'll get off my soapbox, but I really wanted to add that because I think that's just such an important point. Again, I get back to my two golden rules, serve authentication TLS. This is on Azure. Serve authentication is TLS. Client authentication is managed identities. And even if you're using a process that requires some other kind of credential, then govern access to that credential using managed identities. I'll get on my soapbox now. 100%. Do it the safe way.
So I've stayed pretty quiet for this episode, but which is unusual for me, but I think you struck, how do I put this? Struck a nerve with Michael and Mark there, so I wanted to let them say their things, Joey. But at the end of our episode, we always ask all of our guests for their final thought. Like if you're gonna leave our listeners with one thing, what would it be? Turn on MFA. Is that it? Is that it? Well, you'd think it would be that simple, wouldn't ya?
And then you see attack after attack, after attack, after attack. And try to use phishing resistant MFA or other techniques, right? Stay away from SMS. Use number matching. We talked a little bit about how there was a bit of news out where Microsoft's going to be turning that on by default for folks in the near future, where we use number matching. MFA is not the end all, be all for everything if it's not configured correctly.
But when it's configured correctly and it's used, it can protect a whole hell of a lot. Well, that was a short and simple final thought, which I always appreciate, short and simple final thoughts, because the whole point is that they're easy to remember, so thank you very much for that. So with that, let's bring this episode to an end. Jerry, thank you so much for joining us this week and to all our listeners out there. We hope you found this useful as well.
Stay safe and we'll see you next time. Thanks for listening to the Azure Security Podcast. You can find show notes and other resources at our website, azsecuritypodcast.net. If you have any questions, please find us on Twitter at azuresecpod. Background music is from ccmixter.com and licensed under the Creative Commons license.