Welcome to the Azure Security Podcast where we discuss topics relating to security, privacy, reliability, and compliance on the Microsoft Cloud Platform. Hey everybody, welcome to episode 29. We have a full house this week. We have myself, Sarah Mark and Gladys. We also have a guest, Minnie Wallyer, and she's here to talk to us about Azure Data Explorer. Before we get to Minnie, let's take a moment and go through the news. Mike, why don't you kick us off?
First thing is the long awaited and took a lot of work. Cyber Reference Architecture, the Microsoft Cyber Reference Architecture, affectionately known as MCRA, is out. It's released. So AKMS slash MCRA is out. So it's got the original capability one that everybody's familiar with, or many people are. We had like 80,000 downloads in the last version, so plenty. But we also added pretty much all of the sort of dense complex, bring it all together diagram.
So Azure Security, native controls, like what are the stuff that's built into Azure and to our cloud that you can use to protect your Azure end-to-end, everything from the user accounts to the devices, to the backend resources, to the apps, to the IoT and OT devices that connect to it. So we've got that. We've got security operations or SOC reference architecture, zero trust user access.
We've got some kill chain stuff in there, a people diagram, so how the roles and responsibilities fit within an organization and work together and what are the jobs to be done. And a whole lot of zero trust stuff and some other security operations and threat intelligence. So pretty much kind of a best of cornucopia of technical goodness. And so that one is, it's out there. It's available, ready for download. I'd love to get any feedback on it. So hit me up on the socials or whatever for that.
Second one is, because the reference architecture is a nice kind of architectural level view, kind of middle of the security org from a top bottom perspective. And then we realize that a lot of organizations are kind of looking for guidance sort of from a top down perspective and how do you organize your program and set up goals and metrics and budgets and all that kind of good stuff. And also how to interact with the business.
And so what we did in our cloud adoption framework was we added a secure methodology. And so just like we have strategy plan, build, manage, organize, et cetera, we now have secure. So as organizations go to the cloud, this is now a native component of it. This is how to do the security part. And so we cover a lot of stuff in there.
There's kind of like a top half of it, which is the risk insights and how do you align to the business and the initiatives and the priorities and the risk registers and language and prioritization of risk. And then security integration, kind of how do you do that deeper in the org, business resilience and how to think about that. So these are the things that essentially security provides to the business and what the business should expect of sort of the CISO and team.
And then we also have kind of the bottom half of the lower half that's focused on the security disciplines. And we aligned it to NIST, but a little bit more closely actually to the open group where we wanted to have like really actionable specific disciplines that are both familiar but also push the organization into the future. And so access control is the first one. And that's really where networking and identity really need to come together.
Not that sometimes they're on the same page, but oftentimes not in an organization, but really to kind of have this sort of end to end view of how do we actually provide access in this age of cloud, security operations, how do you handle the incidents that are coming in, the active realize risks, asset protection, how do you think about this in sort of a dynamic environment with things popping up in infrastructure as code all the time, new services, new SaaS apps.
So how do you really think about that and do it right? Security governance. And we kind of made governance kind of fun. And I know it's a little crazy to say that, but we did kind of help bridge it to the business, talked about how to deal with like continuous change. And what are the kind of key hallmarks of success there and kind of how to think about that program. And then we also have innovation security, which is really getting into the DevSec op space.
And, you know, what does good look like there? And there's some cultural elements to it. There's some technical elements to it and some process pieces. So definitely check out the CAF secure stuff. Links are both convenient to show us. Okay. So time for some news from me. I'm going to go for not my baby just to start with.
By the way, I've now been having some emails from some of the people out there in the big wide world actually referring to Sentinel as my baby in emails, which I find hugely amusing and thank you very much. And I should probably stop referring to it as such though. Anyway, let's talk about Azure Security Center. So there's some cool things that have come into preview and GA this month. So for a start, we've got a new resource health page that's in preview.
So that's not something that we've had before, but now it's much prettier. It has a nicer view. So go and check it out if you're using Azure Security Center. And if you're using Azure Defender, you can also now see on one page the outstanding security alerts, which is pretty neat. The other thing that in preview is using Azure Defender for Kubernetes. Now that's been around for a while, but now you can use it to protect hybrid and multi-cloud Kubernetes deployments before it was just using AKS.
So if you've got Kubernetes running in another cloud or you have it maybe on-prem, I mean you could have it on-prem, I guess. You can now, using Azure Arc, actually have that protected with Azure Defender. So again, very cool. We've also got some more recommendations around Azure Defender for DNS and resource manager. So if you're using Azure DNS, we've now got Azure Defender for DNS and we've got Azure Defender for resource manager.
So of course, if you're using Azure Defender, go and look at turning it on. There are charges, so do have a look. I think it's always important to say that. Something else that's just cool for my part of the world is that they've also added some new regulatory compliance standards into Security Center. So if you're unfamiliar with this feature, it basically allows Security Center will rate the infrastructure that it has, services it's tracking.
It will rate it against regulatory compliance standards. So we've had a lot of ones unsurprisingly focused on the US and Europe to start with. So NIST, we've got HIPAA, et cetera. But just yay for my part of the world, they've now added the New Zealand ISM, the Information Security Manual. So for those of you in New Zealand, you can now go and rate things against that government standard.
So just because I'm focusing way too much on New Zealand, we've also got the Azure CIS, the Center for Internet Security Benchmarks, and CMMC level three. That's probably my interesting things for Security Center. So moving on to another thing that I'm a big fan of is AKS. So AKS now has support for the secret store CSI that's gone into public preview.
So that means that if you're using a container storage interface driver, you can mount secrets, keys, inserts stored in your secret stores into your pod as a CSI volume. So that just basically means that it's going to make your secure access to secrets much, much, much more straightforward. And you can do it via the containers file system.
And then we've got to go on to, of course, my baby, something that one of our colleagues here in Microsoft has created is the Azure Sentinel Sock Process Framework, and that's been created by RIN. Now he's actually going to come, he's going to be a guest on the podcast to talk about this in lots of detail in a few episodes time, but definitely go and check out what he's done.
He's basically created this whole Sock Process Framework workbook that will show you what you're doing, what you're missing, things that you need to add into your Sock to make it a good, mature process. And it's really, really interesting stuff. I know that he's also been doing it with some of the other great people who work on Azure Sentinel throughout the world. And so definitely go check out his blog post. It's now also just because I did it myself.
It's just been added to the official Azure Sentinel repo. So in the not too distant future, you should actually see the workbook in the Sentinel UI, but if you want to look at it before then, and I really recommend you do, have a look at the link to the blog post that we have in the show notes and go and check it out because it's a very impressive piece of work. And I'm really looking forward to talking to RIN about it in a few episodes time. And yeah, that's all of my news.
So over to you, Michael. A whole bunch of things caught my interest over the last couple of weeks. The first is an announcement that came out of Microsoft Build this week, which was Azure SQL Database Ledger. Probably one of the best ways of thinking about this is imagine if you took a SQL database and add basically kind of the power of blockchain to a SQL database. This allows you to have records in the database that you can show have not been changed.
Or if you do make an update, you can show the complete record of the changes that happen in the database. This is really fantastic for people for say regulatory requirements where they can show the chain of custody, show the lineage of data changes within a database. The next one is new security features in Azure VPN gateway. So for example, you can now have multiple authentication types on a single gateway for open VPN tunnels.
So you could have Azure AD, you can have certificate based and radius based authentication all on the same gateway. There's also border gateway protocol diagnostic support in there. And we've also added VPN packet capture and much better VPN connection management as well. So a lot of good improvements that I know a lot of customers have been asking for. We've also added a new training class to help people learn using Bicep. So Bicep is just essentially the next generation of ARM templates.
It's currently in preview and we're taking a lot of suggestions from our customers. Do be aware that if you're going to use Bicep that there is a really good chance that there will be breaking changes. But Bicep is essentially a more modern, more clean way of building templates for deploying infrastructure as code. We've also added some enhancements to Azure backup. Just notably we now have support for managed identities, for managing permissions on keys that are used in recovery services.
We've also now enabled encryption using customer managed keys during creation of the recovery services vault. That is in limited preview. If you're interested, there's a link in the show notes through the article that has the email account that you can email to opt in for using or kicking the tires on this particular feature. And finally, there's now been added new Azure policy support to enforce encryption backups using customer managed keys.
We've also added some new security features to API management. Most notably, there's now the ability to validate a client certificate. So if you're using TLS, which of course you should be, which will give you server authentication, and you opt in for, you can now opt in for using a client certificate for authentication as well.
We've also added a update to the ciphers and protocols page inside of the Azure portal that makes it much easier for managing cipher suites and protocol versions when using TLS. We will also warn you about using weak cipher suites or weak protocol versions. The last one is Azure key vault is now updated at service level agreement. It was historically 99.9% SLA.
We've now bumped that up to 99.99 SLA, which is fantastic for those people who are using Azure key vaults in mission critical environments that require a high level of SLA. So that's all I have for the news. Glad it's over to you.
I wanted to talk about CMLAM, which is an open source initiative to help security researchers deploy labs in where they can reproduce well known techniques used in real attack scenarios in order to test and verify the effectiveness of services like Microsoft 365 Defender, Azure Defender and Azure Sentinel. I haven't had time to play with this yet, but I hear from coworkers that is awesome resource. In addition, I wanted to talk about renewing release of Azure Sentinel sub process framework.
As part of this podcast, we have talked about many of the threat protections and security operation cloud services that Microsoft has released and how they help streamline the protect the text and respond process.
While these services provide many capabilities, I have seen instances where organizations do not take full advantage of all the capabilities provided by these services because their organizational process or procedures that they use in different in share with different team have not been updated.
For example, I was hoping an organization that had Microsoft E5 threat protection suite, but then the tier one analysts were using some manual processes and other tools to construct reports that they were going to provide to their leadership. When I asked them why they did that, they mentioned that their processes dictated the tools and report format that they needed to use.
Although our services provided more detail and more information, the only way for them to reproduce the reports they wanted in a fast manner was to use those other processes. In essence, this process framework that we released provides organization with an initial set of tasks that should be performed to ensure that issues like these are avoided.
Also it helps the organization to know what are the processes that they need to be looking at, maybe readiness, hopefully take full advantage of the investment that they have made. There was a live presentation of the capabilities that RIN provided, but if you missed it, you could either search it in YouTube or I think the first week of July, RIN will be with us. So be in the lookout for our podcast during that week.
Last, I wanted to mention about the IoT Hub Service APS support for Azure Active Directory based access control. Basically, you now can grant specific service API access permission to user service principles and manage identity from your Azure AD tenant using Azure RBAC or role-based access control to get started just grant roles with the new permissions.
In the links that we are providing in our site, you will find some information about these building roles, permission and samples that you could use to implement this. Let's now turn our attention to our guest. We have Mini Walia. She's a senior program manager in the Azure Data Explorer product group. She's here to talk to us. Funnily enough, about Azure Data Explorer, many welcome to the podcast. Would you like to spend a moment explaining what you do at Microsoft?
How long have you been with the company? Thanks, Michael. Thank you for having me on this podcast today. I'm really excited to be part of this series. My name is Mini Walia, and I'm working as a program manager with Azure Data Explorer product group in Microsoft. It's been almost four years with Microsoft. In this role, I work very closely with large enterprise customers to build secure and scalable big data analytics solutions that are mainly focused on telemetry data.
I'm fortunate enough to work with some of the best Microsoft engineers to continuously evolve, improve and introduce new capabilities in our product. My first question is probably the most obvious one. I'll be honest, I had the same question when I first started learning about this. What is Azure Data Explorer? Azure Data Explorer, the short form we call it as ADX. It's a fully managed big data analytics platform.
It's a distributed columnar store that is mainly purpose-built for real-time analytics over telemetry data. Let me clarify. When I say telemetry, it's a broader term that covers any type of logs, data coming from IoT devices, sensors, connected vehicles, or clickstream data, or it could be any type of events and user activities. Obviously, this is a product that is designed to ingest significant amounts of data. What runs underneath this? It's a Microsoft's proprietary database.
Think of it like an append-only analytical database, which is a bit different from the typical general-purpose databases or transactional databases we have, for example, SQL DB or Cosmos DB. In a sense, I would say it's not really meant for transactional scenarios or frequent update to lead scenarios. It's built for analytical append-only workloads to build low latency, high throughput, near real-time analytics dashboards.
Mini, when we were talking about this before we started recording the podcast, we were talking about the scenarios for using ADX. There's roughly maybe about three main scenarios you see at a high level. Could you talk some more about those? Sure. There are three broader scenarios which are meant for using Azure Data Explorer or, as I said, ADX short form. The first one is around telemetry analytics. That is what it is purposely built for.
It's mainly focused on log analytics and time series analytics scenarios wherein you can do interactive analytics and Hock explorations of data and build near real-time dashboards. The second broader scenario would be around advanced analytics. ADX powers data science and machine learning workloads with a lot of native capabilities for pattern recognitions, forecasting, anomaly detections, those advanced scenarios.
The third scenario is wherein a lot of customers and ISVs, they build single or multi-tenant SaaS solutions using ADX similar to what Microsoft has done. For example, Microsoft has built various SaaS solutions on top of ADX in different domains, like we have got in the monitoring domain, we have Azure Monitor. In the security domain, we have got Azure Sentinel, Security Center, Advanced Thread Protection.
Same way, we have got different products in the IoT domain, time series insights, Play Fab in the gaming domain, so on and so forth. There are heaps of products which are built on top of it. In nutshell, three key broader scenarios, telemetry analytics, advanced analytics, and SaaS solutions which you can build on top of it. Well, I'm going to dig into one of the things you mentioned there.
Obviously, everybody who listens to this podcast knows I have to bring it in somehow, but you mentioned log analytics. Could you explain that? Of course, log analytics sits underneath Azure Sentinel, but what's the difference between ADX and log analytics? Log analytics is basically a SaaS solution that's built on top of ADX. ADX is a PaaS solution, the underlying platform on which log analytics is built.
Log analytics has got out of the box capabilities or I would say domain knowledge in the infrastructure monitoring space. As ADX is an underlying platform, so it brings in value in terms of providing full flexibility and full control on data. When I say data, it could be the management of data, data schema, so the customers get full access to this underlying platform. You can do powerful native analytics on top of this telemetry and time series data. It's a very cost effective platform.
To continue Sarah's thought, these are obviously two different products as you data explorer and log analytics. Do they bring different value propositions to customers? Absolutely. As I said, the key value proposition ADX brings in is in form of bringing in full flexibility, full control and customers can go to the granular level of managing their data. For example, they can even manage till the role level security of their data. The best part of ADX is its price performance ratio.
It can be used as a hybrid solution. Customers mix and match. They can use log analytics as well as ADX, a hybrid solution to get the best of both the worlds. Think of it like Microsoft has built SaaS solutions on top of ADX and we are providing full transparency and full control even on the underlying platform. You can customize the way you need. I think that's a really important point actually. I really want to make sure I listen to understand this. We have this tool, log analytics.
We have tools like Sentinel and so on that feed into log analytics. But ultimately, if you want absolute access to the low level data for your own analysis work, then that's provided through Azure Data Explorer. Did I get that right? Absolutely. Spot on, Michael. You got it absolutely right. I have another question, Mini. You mentioned that ADX is something that we use internally. Can you talk more about what Microsoft does with ADX?
Sure. As I said, ADX is used internally, heavily, and that's how we started. Let me give you a bit of a background around the journey, how we started. In 2015, Microsoft started using ADX internally, heavily for collecting its telemetry data from a lot of services, including Power BI ecosystem, SQL servers, Windows, and there are many other systems that are sending its telemetry.
Just to give you an idea, for example, think of it like every VM in every Azure Data Center globally is sending its telemetry to this platform. We are collecting more than 40 petabytes of data every day onto this platform. In total, it's dealing with 2.5 plus exabytes of data. That's the scale it works at. You can imagine how Microsoft is using it internally for handling its massive petabyte scale of telemetry.
After seeing significant success internally, we made it available to our external customers and the service went GA for the external customers in 2019. Cool. It's very impressive stuff. If any of our listeners are thinking, oh my God, I want to use ADX, how would a customer go about starting to use ADX? Sure. It's super easy. It's simply just like any other service on Azure.
You go to the portal, create your Azure Data Explorer cluster, and you can provision cluster either via portal or through the ARM templates. That's it. Once the cluster is created, you create your databases, tables underneath, and you can access the cluster via the tools we provide. There is an ADX web UI, which is very easy and user-friendly tool. You can use to ingest the data, for example, with a single click, run your interactive analytics queries, build your dashboards.
All those capabilities are baked into that ADX web UI. We do have Thick Client as well, but most of the customers, they prefer to start with the ADX web UI. When would customers use Log Analytics and ADX together? Are there any scenarios that come to mind? A lot of customers who are building hybrid solutions using both these platforms, the key scenario is when customers are dealing with massive amounts of data. For example, Microsoft is using ADX for handling petabyte scale of telemetry data.
Same way when customers want a viable, scalable, and cost-effective solution. Plus, when some of the requirements need customizations or full control on data, that's when they use ADX. ADX becomes a centralized repository for all of their data. They do take value which Log Analytics brings in in the monitoring space, or Sentinel brings in in the security domain. They use ADX even for longer retention of data.
For example, they use Sentinel for 90 days of immediate analysis and detection of their data. They route data past 90 days to ADX for longer-term retention for audit or compliance or other reasons. That's another scenario which they use it for. It's silly me asking the question because I've been working with a customer in this exact space just recently. The conversation went like this, hey, it's going to cost us a lot of money to store all this log data. What log should we store?
Unfortunately, when it comes to security information, you want to store everything because you never know a few months down the track an incident may happen. If you didn't have the log, then you can't do the forensics. They're basically ingesting absolutely everything, not just a cost perspective, but also from a minimal technical depth perspective, they're looking at using Azure Data Explorer, exactly for this.
They're just ingesting absolutely everything, putting it into Azure Data Explorer, and by far the cheapest way they found for storing just these incredible amounts of data. Absolutely, yes. There are a lot of features which are available. For example, we do support cross cluster querying. Customers can ingest data into Sentinel, which uses log analytics workspace under the hood. ADX is the underlying platform for both of these products. They can build their federated queries.
They can enrich the data by building the federated queries via these cross cluster querying capabilities. They can, for example, create dashboards where they can bring in the data from both these platforms without moving data here and there. There is a lot of flexibility, as I said. Is there anything you've talked about a lot of cool tools and features, but is there anything that's coming up that's exciting in ADX LAN that you're able to tell us about?
We keep evolving, improving, and keep adding a lot of features in our product. The hottest thing which is coming up is the integration of ADX within Azure Synapse Analytics. Azure Synapse Analytics is a single stop shop solution for the customers for their big data analytics solutions. ADX integration within Synapse fills in the gaps for managing and for dealing with the log analytics and time series analytics scenarios within a single umbrella. That's what we are working on.
It's in private preview at the moment. If anyone is interested, they can reach out to us. Another cool thing we are working on is contributing onto the open source solution, Telegraph. Telegraph is an open source solution for collecting metrics, and we are building ADX output plug-in for Telegraph so customers can customize and ingest data using n number of input plugins that are available within Telegraph space. These are two cool things relevant to the monitoring space.
Minnie, if you've got one thought to leave our listeners with, we ask this to every single guest we have on the podcast, what would it be? So, Sarah, I would sum it up by sharing a quote from one of our customers. So when we made ADX available for our external customers after seeing significant success internally, the customer after using ADX, they shared their experience by saying ADX is Microsoft's best kept secret.
It's very mature and I would say it's a battle-tested platform over petabyte scale. As I mentioned, how Microsoft uses it for ingesting 40 plus petabytes of data every day. And the last thing I would say is it's super easy, very user-friendly database, and it comes up with tools and, for example, ADX Web UI I talked about and we support Kusto Query language, which is, again, very, very simple and user-friendly and SQL is also supported.
I would encourage all the users to try it out and we are always keen to hear your feedback to improve our products. And with that, let's bring this episode to an end. Minnie, thank you so much for joining us this week. Just for our listeners out there, Minnie isn't feeling very well, so thank you so much, Minnie, for coming along and joining us. To our listeners out there, I hope you found this podcast useful. I know I definitely did. I always learn something and this is certainly no exception.
So to everyone out there, stay safe and we'll see you next time. Thanks for listening to the Azure Security Podcast. You can find show notes and other resources at our website azsecuritypodcast.net. If you have any questions, please find us on Twitter at AzureSecPod. Music is from ccmixter.com and licensed under the Creative Commons license.