Welcome to the Azure Security Podcast where we discuss topics relating to security, privacy, reliability and compliance on the Microsoft Cloud Platform. Hey everybody, welcome to episode 80. This week it's just myself, Michael and Sarah and our guest this week is Matt Zorich who's here to talk to us about a team at Microsoft called DART and Microsoft Instant Response. Before we get into the discussion with Matt, let's take a little lap around the news. Sarah, why don't you kick things off?
Okay, well, you know me and my news, I always love to talk about something related to my baby, Microsoft Sentinel. This is tangentially related, which is Azure Monitor logs now have better table level RBAC. So you may remember if you've tried to use table level RBAC in Azure Monitor, it hasn't been the most flexible and it's had some restrictions on how you could use it. But we now have public preview for much better and much more granular RBAC.
And it will also allow you to do it for custom tables, which is something we couldn't do before. So we'll have a link in the show notes. If that's something that you looked at in the past and it didn't quite work for you, go and have another look because it's definitely much improved on what it was. And that's it for me this time, Michael. So over to you. Yeah, I've got three items. They're actually all sort of somewhat related. They're all to do with web application firewalls at some level.
So the first one is in public preview, we now have sensitive data protection for app gateway web application firewall logs. So you can enable this and certain things that may be deemed a little bit sensitive. So for example, JSON argument names, request IP addresses, request header names, they will be sort of scrubbed with a whole series of asterisks. So again, things like IP addresses, what could be potentially passwords, those kinds of things, they can be all sort of erased.
So that way there's loads of chance of PII or other kinds of sensitive data appearing in the web application firewall logs. Next one is also in public preview. We have the default rule set 2.1 for regional WAF with application gateway. So there's a whole bunch of rules that we have that are available for the web application firewall and the different versions. So now what we've done is just sort of standardized on a whole bunch of different types of rules.
And this one is version 2.1, so includes things like cross-site scripting, includes things like well-known JavaScript attacks, remote code execution, session fixation, SQL injection right in my wheelhouse, those kinds of things. So these are the default rules. I mean, obviously you can go and change them. We want to sort of continue sort of raising the bar in terms of core security. Otherwise if I said there was three items, that's only two items allied. So there we go.
So with that, let's turn our attention to our guest. As I mentioned at the beginning, our guest this week is Matt Zorich, who's here to talk to us about the team at Microsoft called Dart and its role in incident response. Matt, welcome to the podcast. We'd like to take a moment and introduce yourself to our listeners. Yeah, thanks for having me. Actually, yeah, to clarify, just like a lot of things at Microsoft, we have also gone under a rebranding.
So we are now Microsoft Incident Response, previously Dart. And yes, I'm a consultant on the Microsoft Incident Response team and I'm based out of Australia with Sarah and basically we're Microsoft's customer facing incident response team. So if one of our customers is compromised and they feel the need for a full investigation, then we can be brought in to do that.
And we'll do the investigation and some hardening work and kind of uncover the story of the attack and try to give the customers some good advice to hopefully not be a return customer. We always say we're the one team at Microsoft that can say, we don't really want to get your call ever again. I find the stuff that Dart does fascinating. And I know you're not able to talk about specifics or mention specific customers for obvious reasons.
Tell us about, obviously, this is very fluid and changes all the time, but tell us a bit about some of the trends that you're seeing in your incident response land at the moment. What are the call outs from customers that you're seeing a lot of at the moment? Because I'm sure if I were a listener, this, well, I like to hear this from Dart and I'm going to keep calling you Dart. I know you're not Dart. You're Microsoft incident response.
But I am always fascinated by the kind of things that you folks do and what you see. So yeah, what are the trends? What are the things you're getting called out for a lot at the moment? Yeah, you're right. It's kind of fluid and I think trying to understand why that's the case. Certainly there's no reason why it changes or moves over time. It just so happens to be the engagements that land with our team or things going on.
Certainly in the last couple of months, we've been engaged in a lot more kind of cloud engagement. Certainly I have. So when we do investigations, they may be on-premises focused or they may be cloud, so kind of Azure and Azure Active Directory and Microsoft 365. From my last few months, they've certainly skewed towards Microsoft 365 and Azure. So I'm seeing a lot of those kind of come across my way. What the reason is, I don't really know. It could be any number of things.
But I think we're, you know, to kind of guess, I guess I would say or try and kind of hypothesize about it is I think, you know, maybe post-COVID, now we're in this hybrid work kind of work style again. We still have a lot of people working from home. So kind of the first entry point they might have to your data is like a Microsoft 365.
So maybe ATT&CK is a kind of understanding that people are working at home, maybe not protected in the same way that they would be in a traditional corporate network. So they're kind of attempting to fish or compromise like a cloud user and then kind of pivot from there. But, you know, we see all kinds of ATT&CK still. So we still see a lot of ransomware. We see a lot of like data exfiltration and things like that.
But they certainly appear to have skewed a little bit more towards the cloud in the last few months for whatever reason. So you said that you're seeing a lot of incidents in cloud. Can you talk a little bit more about the kind of incidents you're seeing in cloud and M365? Yeah, yeah, certainly we can talk to kind of the broad tactics that we see. So look, I think a lot of people know if you read kind of any incident reports is that often like large scale compromise begins with a regular user.
And that's certainly what we see reflected. So it might be like a non-privilege, just a regular user that gets fished or they get kind of the MFA bombing that you hear about or, you know, they click on a dodgy link or they have a weak password. However, kind of the attacker gets that foothold. And then what we're seeing is them kind of use the M365 kind of platform and just search within a user's mailbox. So they might search for other credentials.
They might search for VPN information and things like that to kind of progress their attack. I think ultimately towards the end of the attacks in the cloud, we're seeing kind of a mixture of things. So it can be very financially motivated. So, you know, if an attacker can take over an Azure Active Directory tenant, they might then look to pivot into Azure and spin up kind of crypto mining infrastructure and mine crypto on the customer's dollar.
Obviously the customer gets left with the bill so that can be very quick and very kind of expensive. The other one we're seeing is kind of the data exfiltration and extortion type of attacks where an attacker might get full tenant level access to everything that's in SharePoint and OneDrive and Teams and everything else and kind of take copies of that data and then try to extort the company saying, you know, I've got X amount of data, pay us this amount before we leak it.
So they're kind of the two strains I would guess we see that the crypto mining and the extortion in, you know, with that data theft. For those two strains, are you able to tell us a bit about kind of how you would deal with them? Yeah, certainly.
So look, the crypto mining side of things is usually quite easy to detect because generally the customer gets an alert or something saying you might have a huge bill that's been rung up because I think the crypto mining kind of the people that are skewed towards crypto mining know that they're on a very short time frame that they're going to be detected and they're going to get kicked back out. So really it's kind of like a smash and grab.
You know, we might only have access to this this tenant for a few days. So we're just going to spin up as many VMs as we can to mine crypto before we're back out the door with the extortion and those kind of more sneaky tactics. What we try to do when we are first engaged with a customer that's kind of in the middle of that is try to take back control of the tenant from kind of a global administrator level. And we call that kind of positive identity control.
So we want to make sure that all the global admins, they're under our control, that there's no persistence mechanisms left behind so the attacker can get back in. Once we're kind of happy that the environment's stable and it's in our control, then that's when we start doing kind of the deep dive and you know what did the access, sorry, what did the actor access? Did they access data? You know, have they created accounts to get back in later and things like that.
But yeah, the first step in these kind of compromises is always try to, we call it positive identity control because someone's always in control of your tenant. We just, we prefer it's you that's in control of your tenant, not the bad guys. And we kind of pivot from there and really with these investigations, it's where the breadcrumbs lead us because no two investigations are the same. And look, we certainly see the same tactics and the same motivations, but they're constantly evolving.
There's always novel ways and novel things that they're trying. So we just try to get all the data. We talk to the customer. We try to get logs. We try to get information and it's a bit like a detective story, right? You're just trying to piece together what happened. You're also trying to, it's always hard to distinguish between sometimes threat actor activity and users doing silly things activity as well. So often we have a lot of questions for the customers to clarify those things.
It's interesting you just say, you know, data X fill and Bitcoin mining or some kind of digital currency mining. I mean, that's the result, you know, the end result, right? Someone obviously got into the system and so on, but it's interesting you just say things like fishing or clicking on a bad link. It's almost like what's old is new again. I mean, how long have we had these kinds of attacks? And we're still seeing them today as the way the bad guys get into the environment, right?
I mean, so I mean, am I kind of right in that or are we seeing updated variations of these kinds of attacks or is it really just the absolute basics? It's a bit of both. Yeah. Look, you're certainly seeing some of the, yeah, look, people still click on fishing links. That's the reality of it. But we are seeing some more kind of novel and modern attacks, so certainly things like social engineering are becoming more popular.
So you know, an attacker might email the help desk of a company and pretend to be a, you know, a user of that company and say, I've got a new mobile because I lost my old one or my old one broke and here's my new number. So can you update it in this system? And then the attacker can use the self-service password reset function to kind of take control of that account. We're also seeing, you know, you see things, not just traditional fishing, but you might see smishing.
So, you know, SMS is sent to users and credentials lost that way. So it's still true. It's still fishing, I guess, but there are some modern kind of takes on it and things like that. But for social engineering, that's the hard one to kind of protect as well because you're relying on your help desk and your IT teams to kind of, it's a very soft skill to understand whether something's not quite right and kind of challenge users and things like that.
So it's, I appreciate it's very hard to stop as well. So when you said smishing, so using SMS as an attack vector. Yeah, so is this because I'm thinking of multi-factor stuff, right? Something about multi-factual authentication using just SMS messages. Is that the attack we're talking about or are we just talking about in general, hey, this is Bob from support. Please log in and click this link and change your password.
Because the big thing in multi-factual authentication is not to use SMS, right? It's to use an app instead or something like that. Yeah. So it tends to be the latter of that, which is, yeah, send some kind of message out, which is, you know, update your payroll information or this is IT security and can you confirm your details or whatever they look like. And then to combine with that, you might get some kind of social engineering to get through the MFA process.
So they might call them up and say, you know, this is Bob from the help desk or it could be they often do like what we call like kind of spamming different MFA methods. So they might send a message and then they might do a prompt and then they might do a phone call and et cetera, et cetera, until one kind of gets through. And once they've taken control of that account, they can put their own MFA method in as a persistence mechanism. So they only really need to get through just once.
And then, you know, they register their own phone or their own authenticator app. It's the law of large numbers, right? It's like if you hit enough people, someone's going to click on the link, right? A very small percentage of a very large number is still a large number. So that's what's working in the attackers advantage, right? They don't have everyone. Not everyone has to click on the link or respond to the text message or respond to the email. All it takes is one person ultimately.
So it's the law of large numbers. You know, again, a small percentage of a very large number is still a large number of potential victims. Yeah, definitely. That's what I always say. I always say like being a defender or a blue teamer is really the hardest job in cybersecurity. Like I personally believe because you need to be perfect every day. The attackers and the bad guys really get as many shots as they want. It doesn't matter if they miss. They just like you say, more messages, more emails.
Eventually someone's going to click. The law of averages says they'll get someone eventually. And that's like, you know, you laugh at some of these emails and you'd be like, oh, why would anyone ever click on them? But it doesn't cost anything to send emails. You know, someone will click. I think I should bring that up about, you know, essentially being the defender.
The very first book that I ever wrote when I was at Microsoft was called Designing Secure Web-based Applications for Windows 2000 back in the day. And in there I actually wrote a small section called The Attacker's Advantage and the Defender's Dilemma. And that's exactly the problem, right? Is the attacker just has to get lucky once. The defender has to protect 100% of the time and be correct 100% of the time. The attacker just has to sneak in once, right?
There's one little chink in the armor and there you go. Yeah, exactly. And the defenders often have to deal with the reality of business, which might be, you know, they don't want to change. They don't want to deploy MFA. They don't want to do things that inhibit productivity. And they have these, like the defenders have all these corporate rules to follow and budgets and things like that. Attackers don't have any of those. There's no scope of work they can really do whatever they like.
So that makes it doubly hard. I often hear people say, well, I'm not a really big, well-known, you know, Fortune 500 company. Why would someone try and attack me? Why would someone try and hack me? And I just wondered if you had any thoughts on that. You know, the types of customers that you actually end up helping, are they much bigger, high profile, well-known brands?
Or do you go to a lot of, you know, smaller customers that might be the type that would say, hey, why does anyone care about me? So I think, yeah, no one's definitely no one's immune from cyber attack. For sure. I can definitely tell you that. Like whether it's small business, medium business or massive enterprises might be, you know, the attackers might be motivated by different things depending on the size of the organization.
Our team probably skews towards bigger customers, but that's probably more because, you know, we're Microsoft and we're a big IR team and we kind of play in that market. But there's plenty of other cybersecurity incident response firms that kind of are engaged with small, medium businesses as well. I don't think, like I said, I don't think anyone's immune. And a lot of the attacks are opportunistic.
So if they, like you said, they send out all these phishing emails or smishing and they get credentials for a particular company and they're able to access it, that might be the reason to go after that company rather than being particularly targeted. So yeah, every range. And I think the ones you read in the news and things skewed towards the bigger companies as well, but you're likely not hearing about the smaller and the medium sized enterprises that are getting attacked.
It's just kind of not as newsworthy, I guess, unfortunately. But you look at the stats for things like business email compromise and business email compromise is not flashy.
It's been around forever and that's simply compromising someone's user account and logging onto their email and then you might change the invoice on an email or something and to your bank account details, which sounds like they might be $20,000, like a $20,000 invoice, which doesn't sound like much in the scheme of things, but the stats for things like business email compromise, if you look at them, it's wild.
It's billions of dollars that costs every year and $20,000 to a massive company is a drop in the ocean. And that could be devastating to a small or medium business that's kind of month to month. So yeah, like I say, no one's immune. What things would you recommend based on what you've seen that people should focus on in terms of security controls to reduce the likelihood of a breach?
I mean, I know that in a compromise, I mean, I know that's what we all do every day, but I'd be keen to hear from an incident responder. What do you think the things are that people should really focus on? Yeah, definitely. It's going to be a very, very boring answer. Honestly, it's the basics. And like I always say, don't get the basics confused with it being easy because they're two different things, especially in massive, massive organizations.
I'm not sure if you've ever seen the Microsoft Cybersecurity bell curve. I'm a big fan of that. You know what? I have that nowadays, no joke, in pretty much every presentation I do. We'll link to it in the show notes. It's in the digital defense report. Sorry, Matt, go on. The reality is, is like being an incident responder is you see that it is actually true. It's the basics.
So I think it's like 98% of attacks will be stopped by, you know, MFA, least privilege, attaching vulnerable servers and kind of applying zero trust and having like a modern anti-malware, like a defender for endpoint or something like that. And I remember seeing a phrase once, it was like, basically what we're after is brilliance in the basics. And that's like a term that's always stuck with me and I've really enjoyed it. And I use it occasionally is, and that's what it is.
It's doing those basics, doing them across a whole organization. And like I say, like I've been on the other side and worked as like a blue teamer in big organizations and it is hard, like hundreds of thousands of users or millions of users and deploying MFA to them. Like we understand it's difficult, but there's no secret source. If you do do it, it'll reduce your risk. And that's what we always say is there's no 100% secure. It's risk mitigation.
And if you've got 100,000 users and you can deploy MFA to 90% of them, that's a huge win. Even if you can never get the last 10,000 or whatever it is, it's all reducing risk. What are the common mistakes you see that people make that subsequently end up being breaches? I realize this is kind of sort of very closely linked with what I've just asked you.
Yeah, I think that the constant ones we see in this is both in on premises and kind of in Azure Active Directory is probably too much privilege for users that don't really require it and kind of not understanding what that privilege can grant users to. So you might have like a service account that's in a domain admins group. We see that all the time or a service account that's in global admins in Azure Active Directory.
Those kind of service accounts and things like that are often not secured the same way like a human identity would be. And that's kind of the nature of them. You can't really put MFA and things on service accounts.
And often credentials for those things are left kind of just hanging around in clear text, whether that's in scripts or that's the old passwords.txt sitting in your One Drive or sitting in your SharePoint because they're very if you can find them, if you search for, you know, I've got my passwords list that I go to. And so I search for passwords, then an attacker can also search for passwords and uncover them as well. So that's definitely the most common one.
And I think the other one is basically not enforcing kind of stricter controls on your tier zero and your very, very high privileged account accounts. So like we said earlier, like we appreciate that it's very hard to deploy all these security controls across massive organizations uniformly. But your privileged accounts and your tier zeros and your domain admins and your global admins, that should be a very small subset of your users. And those users should be held to much stricter controls.
Look, I think everyone on this call and probably everyone listening is you'll never stop regular users being compromised. It's just going to happen. Like we said, whether it's phishing, whether it's smishing, whether it's downloading cracked software, that's okay. Like that can be inconvenient for that one user. What ultimately we're trying to do is stop the kind of the compromise of a single user turning into a compromise of all our users.
And that's where we need to, if we harden our tier zero, we can deal with the user installing malware. We can give them a wrap on the knuckles and send them a new laptop. That's no big deal. But we don't want that to be like the first domino that ends up in a huge incident. Just last couple of questions for you, Matt. Hopefully none of our listeners will have to do this.
But if anyone out there did need the services of Microsoft IR, what is the process to get in touch if a customer thinks they need help? Yeah. So there's a few ways. So you can log a severity A case through your unified support. And we kind of on the Microsoft IR side, we kind of keep an eye on those. They don't always require full incident response. Sometimes the cert team can handle it or it's not actually a security thing. It might be an operational thing.
And we'll kind of keep an eye on any that may come our way. Or we believe that the customer may need full incident response because they don't maybe we don't appreciate kind of the situation they're in. We also were available under retainer. So if you're interested in having Microsoft IR just available, it's like, you know, pick up the phone, something's happened. We can kind of mobilize very quickly.
There's also just like on a link on our website, and I'll find you the link just basically, you know, we're suffering a security incident at the moment. And that comes through to our team and we reach out and kind of see what's happening. And again, if you if we feel you need full incident response, and we can try mobilize a team and get going. So lots of ways.
What, in your opinion, what is the value of some kind of protected workstation like a protected access workstation or privilege access workstation, I should say, or, you know, secured access workstation? In other words, people shouldn't, you know, not going into production, like not administering production from their from their normal work machines, rather they're going in from, you know, machines that are designed to do a specific task, which is administration. And that's it.
But you can't do your email, you can't go and look on Facebook or any of your socials. Is that something that that sits well with customers? Or is it a bit of a hard sell? It's one of those things it's we always laugh about it is like the saying is like, you know, never waste a crisis. So post incident or when we're engaged, things like the privileged access workstations and MFA and things like that, they all they all become a really good idea.
It's very it's very easy to kind of invoke change during the incident or in the brief period after. So often customers are very open to that idea because they've seen the ramifications of not not having a privileged access workstation. And it is like a fantastic control. And it lends on to what I said earlier about protecting those tier zero accounts as well as you can. So, you know, privilege access workstation is ultimately trying to stop the kind of spread of privilege credentials.
Because if you think about like if you've got 10,000 workstations and you've got a domain admin and if a user has a problem with their workstation, they log on to the workstation as their domain admin account and kind of help the user out. What you've done is actually left your credentials on that end user workstation and an end user workstation is probably not going to be secured as well as a domain controller or things like that.
So if an attacker was to kind of compromise that end user device, there's a chance that they could compromise the domain admin credentials. So a poor or, you know, a secure access workstation or whatever you'd like to call it basically says if you're going to access a tier zero service like a domain controller, you can only access it from this particular device and it's got its own controls. And like you say, it doesn't browse the Internet. It doesn't have a mailbox so it can't be phished.
It's hardened. It's whatever controls you want to put around it and that way we're not leaving domain admin credentials kind of scattered around the environment in lower tiers. So yeah, it's a very, very strong control. Definitely worth doing and I think it's gotten easier as well in time. I think you've got things like you can have like virtual secured workstations now running in Azure and things like that.
So it doesn't always have to be, you know, another laptop or another device and things like that. Yeah, I did some work a few years ago with a healthcare company and they were moving some of their workloads to Azure.
And basically it was, if you access, even try to access production in Azure without using a privileged access workstation, it was actually a potentially fireable offense because they needed to make sure that, you know, access to the environment was through a secured environment, not from Bob's laptop, you know. Yeah, that's right. Yeah, that's exactly right.
Yeah. And with, you know, things like conditional access in Azure, I do really help that now because it's good to have, you know, the policy of course saying that you can be fired if you do that, but you really, what you really want is like a hard technical control because people will be lazy and eventually they'll probably kind of, you know, if you have an incident and you put a paw and you put all these cool rules in and that's awesome and
kind of 12 months later, two years later, the security, you know, the compromise you had is kind of a distant memory and then the bad habits sneak back in. That's just human nature. Yeah, it has to be a strong preventative control, right? Like not a responsive control. It has to be something that prevents someone from accessing the environment unless they're using a privileged access workstation or similar.
Yeah. Yeah. It's good to know that you're seeing people actually adopt it, but only after that you're like, you know, the incidents happened, but I guess that's better than better than never. So Matt, one thing you may be aware of is that every time we have a guest, we always ask them for a final thought. So if there's one final thought you'd like to leave our listeners with, what would it be?
Yeah, I think my final thought kind of speaks to what we were just talking about actually in that, look, in our experience, we see the don't waste a crisis mentality and like it's great that companies go through and make the changes after the fact, but would really like to have people to do it beforehand. And really it's easier, it's cheaper, it's less stressful to make those changes before an incident than when you're in the heat of it.
And kind of leading on with that is that a saying we have is that don't, you know, don't let perfection be the enemy of good. It doesn't need, your security controls don't need to be perfect. You don't need to be a hundred percent coverage of MFA or it doesn't need to be that. It can be just iterative. So you can deploy MFA to your admins, to your privileged account, to your board members, things like that. And it's all risk reduction.
So trying to kind of architect solutions for a hundred thousand users, there's always going to be exclusions and that's not a problem. But don't let those exclusions kind of get the way in the way of rolling out these things to the majority of users. I think that would be my final thought is like I say, it's all risk reduction. If you can reduce risk a little bit, that's a really good win. Excellent. Thanks for that. So it makes absolute sense. And with that, let's bring this episode to an end.
So Matt, thank you so much for joining us this week. I always learn something on these episodes and this was absolutely no exception. And to all our listeners out there, thank you so much for listening. Stay safe and we'll see you next time. Thanks for listening to the Azure Security Podcast. You can find show notes and other resources at our website, azsecuritypodcast.net. If you have any questions, please find us on Twitter at Azure Setpod.
background music is from ccmixtor.com and licensed under the Creative Commons license.