Welcome to the Talking Security Podcast. We will talk about items related to Microsoft Security. Hello, welcome again to the new recording of the Talking Security Podcast. This time again in the Defender for DevSecOps series and we're back with Pouyan and Sander. What are we going to do today? Today we're going to talk about the build section of DevSecOps. So last episode we talked about code and this episode we're going to talk all about the build environment and build process.
How to build such piece of software or some infrastructure for example to enroll that. So let's dive in. Do we have, first of all, do we have anything to announce? It's almost MVP Summit, so we're both, all three of us, we are moving to America for a week. So we will do some recordings there, I guess. Hopefully. We'll take our gear, not this gear, but we'll take the gear with us and then let's see if we can talk to some people over there. So really looking forward. It will be my first
time so hopefully it will be awesome. First time in Seattle. Your first time at the Summit as well? Yes. Oh really? I thought you were gone there for like a million times already. I think I'm the youngest MVP in this room. You're certainly not the youngest person in the room. No, no, no, no, definitely not. But let's dive into the DevSecOps build process. What is it? What things do we need to cover? Well I kind of see this in two different sections. So you have like the build process and you
have the build environment. And I think it's really, really important to talk about security aspect here because in the normal DevOps flow, when someone talks about DevOps they usually talk about CI, CD, like building your software when something changes and then releasing it automatically instead of doing it by hand with FTP or just doing stuff by hand. But with DevSecOps it's all about the security aspect. So how do we make sure that this building
process is very secure? Because it's very easy to introduce security issues with this. Yeah, first, first dive in the process and the environment, but first the process. Let's see what kind of things do we need to cover? Like the secure build, you talk already about security. What kind of aspects around security do we need to look into? Well I think it's good before we dive in into the aspects is to define what we mean when
we are talking about build, right? Because when we are doing build on an infrastructure project, it's completely different than when we are talking about build, when we are about a.NET application where you need to generate stuff. So I think that's initially because after that we can start thinking, okay, there are multiple parallels that we can discuss and see, okay, what are the security challenges on that aspect? Is that a good intro?
Yeah, that sounds good. I think it's a good thing that you're saying, Clique. We're not just releasing source code onto a machine. Also if we're talking about stuff like infrastructure as code or like bicep or stuff like this, you're not sending the bicep files to Azure. You're actually sending the result. So I think the build process could really be summed up as taking the source of a project and turning it into an artifact, turning it into something
that will eventually run on a machine somewhere. And that can take many forms. Like in.NET you're compiling an application with a normal website. You're maybe just minifying it, but the HTML, the CSS and JavaScript stays the same. And with bicep, well, bicep is just turning it into ARM templates in the end. Yeah. I think what we also were discussing on before the start is how the build is also
changing, right? Because with the introduction of the infrastructure as code, for example, what I told you guys earlier, in the early days when we did the infrastructure as code in Azure DevOps, we needed to do build and then create an artifact which we could use in the release deployed, which is now not the case anymore as well, right?
Kinda, kinda. So now you use, yeah, you used to like click ops kind of the pipeline together with your by hand and it worked and it was really nice to begin with, but you didn't really have any backups. It wasn't in your Git repository or something like that. Nowadays, exactly what you say, you had like, well, you had, you used to have like a build section and a release section. And nowadays the modern approach is to use like a YAML file with your
pipeline in it. And in there you define all of the build steps and then you can also define your release steps in there. Usually you do that in a separate stage or in a separate file. So it's nicely separated, but now it's a whole more, it's a, it's a different approach to building and releasing your application. Yeah. So if you focus on the, let's start with the infrastructure as code, for example, if you take a look when we are doing infrastructure as code or we are doing conditional access
deployment through DevOps processes, right? Things, configuration as code. How do you see, how do we see the build stage and do we see any security issues in there as well? When only having infrastructure as code and deploying workplace related items, identity related items like conditional access, for example, can we deploy that in a secure way and how should we do that? So the authentication part you mean, or is it for example, for example,
you need to connect to a certain Azure actor and try the environment. You need to deploy within the entire ID. You need to deploy the conditional access policies. And in this case we need an app rack that app rack needs some permissions, right? How do we secure that optimally? So nowadays we're using Key Vault. Is that the right way or do we store that in another way? So I think what you're going to is like, for this is of course kind of Azure DevOps specific,
but I think you're leaning into like the service connections bit. Like how do we connect our Azure DevOps environment to our Azure environment to make the changes and how do we create a secure connection? It could be that that could be the way, but you can also run a script on a PowerShell script for example, or whatever. So there needs to be a connection and that connection can be established in different ways in my opinion. Yeah, yeah. You can definitely do multiple things. Yeah.
So what you often see is I think at the moment that in Azure DevOps or people, you can create service connection, which you can use in your pipelines for example. But we had also seen scenarios where people hijacked the connection or the deployment process and then got the token out of the connectivity. And then through that they hopped through. So I think that's what you're meaning with the authentication part. But GitHub it's different. I think you
need to store, you cannot define connections. You need to define credentials, which you can use to connect. Not really sure. I think under the hood, it kind of works the same in Azure DevOps where you create like a service connection to Azure. You can like use like an app registration, for example, like a service principle under the hood. But more modern approaches are like using a managed identity or a workload federation, something. Does someone here know the correct
term? Some workload federation. Federated credential. That's something new. And that's really changing it because it's like managed identity, but it's so you don't have a secret anymore. Yeah, something like that. Yeah. And I think using, I mean, you can definitely use a PowerShell script, right? To like in your pipeline, say I want to connect to Azure. And instead of
using a service connection, I'm just going to do it myself. You could, but I think this is going to come up often more in this episode is why would you do it yourself if you can use like a cloud hosted, Microsoft hosted or provider hosted option, like a service connection. And if we want, if we want to use that service connection, for example, what do we need to do to secure that optimally? Because last days I talked to a few customers regarding
phishing resistant MFA that is more user based. From a user perspective, there are multiple
challenges regarding MFA and phishing regard with the MFA token and so on. Is that, is that the same in the area with the service connection and using a managed, a managed identity of service principle, whatever, do we face the same challenges as well in, in creating connections between a DevOps environment, the Git em environment and a backend system, for example, Azure Active Directory or the, the, the build environment where you are publishing your, your software.
I think, I think definitely, I think then it's more like, I don't, if you're using service connections, what I'm trying to say is that then it really depends on how much access you give that Azure resource, right? Like in the end you're using somewhere in Azure is defined a service principle, a user, let's just say it's like a user and that user
is the thing that you're using in Azure DevOps to connect to your Azure environment. And I've seen a lot of cases where then they're like, Oh, it's really difficult to manage that. So we're just going to give that, give that service principle global admin and there we go. And then like what you said, like if you get your bill process hijacked, for example, you're using like a task that you don't know is malicious, then that of course could misuse
that Azure credential and like hijack your systems. I think it's really important that you then know if I'm going to use service connections, if I'm going to use a PowerShell script, how much can I do? Like I want a minimal access. And I thought that we, in the last few months, we saw a few of these, these kinds of cases that a certain user, a developer is hacked and that the password and the multi-factor
token was stolen and then they are into a DevSecOps pipeline or whatever. And then they running the software and then the software with malicious code in it is deployed to customers. So that is, if a user, a developer has access to that managed instance that apprack with a lot of rights in a certain environment, that could be potential big damage could be there. Yeah. Well then definitely in cases like this is of course, how do you say that? Like if,
for example, I was hijacked, right? And I have a lot, I can have administrator rights somewhere, then that is bad, but hopefully at least my credentials are not used in the automated build pipeline. That is hopefully exactly. You see something you shouldn't do. You should definitely have like a separate entity or separate identity that connects to your Azure environment automatically. Of course, if someone has my token, they could
probably go to the Azure portal. And if I have a lot of administrator, I can still delete stuff or delete the code, but at least it could not really be automated that much anymore. And then we're definitely going to have to talk about the, like exactly what you say, phishing resistant MFA, pass keys, passwordless authentication and that kind of stuff. But don't know how well that fits in the build process here. No, no, no. Now we are looking a little bit broader than only the build pipeline, but
let's go back to the build process. What do we need to, we talked to some points. What do we need to cover next? Because we have, we have a built and then I've tried to deploy that built. What do we need to do in that, in that kind of steps? What do we need to do to secure? What do we store the credentials that are using? Are we only using a service connection or what is more within the build? So I think also pointing a little bit to the
last question, I think it's a great topic. And I think if you zoom out on that topic, it's the whole reason why we are having this podcast, right? Because it's not in one process where we can solve that kind of issue. So the whole DevSecOps, it's the idea is to build that checks and controls in every stage and every phase. So for example, let's say we
have that connection and if we have the right permission, we can use that connection. And we have, as you mentioned, our workflow instead of just somebody clicking it through the portal, but code defined stored in our repository. And like we discussed last time, we have our branch policies and repo protection in place. It minimizes the impact, the chance that somebody
can hijack our workflow to build steps into it without checks being done ahead. So I think yes, there are possibilities, and I think there are a couple of big challenges when it comes to authentication side of the build or even release, because you're always working with service principles or user accounts or API keys, which have permanent permission basis. Exactly. Are you talking really a lot like minimizing the chances that your build process gets corrupted, gets attacked?
Regarding on first time we talked about artifacts that we are not using anymore and we're still using it, but not for infrastructure scope. If we look at nowadays, should we use DevOps, GitHub, whatever, and why should we do that in a certain way? What are tools that we can use to optimally use it in the build process and to see how we can move forward on that? I got a couple of notes for this. One of the things, the tools, we're talking about toolings
and how to optimize the security for your build and your build pipeline. There's definitely things that we also talked about in the last episode. We talked about vulnerability scanning. We talked about container scanning. There's definitely still a mention here for code QL, which is checking your code quality and stuff like that, maybe also vulnerabilities. You
want to use those tools and those tools are best used in a build environment. Before you do a deployment to your production environment, there's going to be some vulnerability scanning happening. They're like, hey, in this version of the SQL that you're using in the package, we found a vulnerability and the deployment will be stopped. Stuff like this is really how you can secure your code deployment process. That is definitely
something that with a DevSecOps approach, you want to automate. You don't want someone to say, oh no, let's just take a look at the vulnerabilities. We'll look at the vulnerabilities each sprint. No one actually does that. That needs to be automated. We had a great session on this topic with David, of course, on the DefSecOps, on the Defender for DevOps, sorry. How Microsoft is also helping to bring visibility into that
process of the build and release. For example, what you exactly said is you want to see which vulnerabilities are coming in the code maybe, but also in the configuration-wise. Maybe secret scanning, we know GitHub has it already or the advanced security has it in house. With tools as Defender for DevOps, you can integrate, for example, with GitHub or with
Azure DevOps. Then you step in into the build process and into the PR process. Then we can say, okay, if you create a PR to make certain changes, then this pipeline needs to run and then do some checks before we allow it to be merged. If there are any findings like there is a static password in your code or you are creating for... If we go back to the infrastructure scope, you are doing storage account with weak encryption or with a VM
with a public IP address. You see that kind of configuration or at least which can end up into being vulnerable already into that build process. That will be integrated to Defender for DevOps as well, so in Defender for Cloud. What I really love about the Defender for DevOps is it has a feature. They call it PR integration where the feedback goes back to your PR. As a developer, you see, okay, I have a PR. Oh, on line 13, there was
a static password found. On line 15, I had an adjacent template for a storage account, but I had a weak encryption model configured. As a developer, you don't need to escape your environment for that build phase, but they can bring it back to you. You're talking now about Defender to secure your code and the tools that you're using.
Defender is integrating in GitHub and Azure DevOps. Are they the only tools that we are still using at the moment or are there any other tools that we are using to create a build environment? To create a build environment? To create your build, to create your code. I think there's definitely, I mentioned CodeCube a little bit. Things like SonarCube and SonarCloud
that will also do a lot of code scanning for you. That will also make sure that the thing that you are going to deploy is secure or is at least secure enough according to the policies that you've set up. In the other end, the question is more or less, we are still using GitHub at the moment. I don't know exactly, are you still using GitHub as well? I use Azure DevOps and Azure DevOps Pipeline for work. Both are within the Microsoft ecosystem, more or less general products that we are using.
Are there other tools like GitHub, like Azure DevOps that people in the wild are using to create code? Sure. To deploy their applications, maybe you mean. For example? Yeah, there's things like Jenkins, right? Which used to be GitLab, CircleCI, I believe GitLab definitely. CircleCI is a really famous one. GitLab is an on-prem environment. I don't know, it's also served. It's also hosted for you. It's hosted, but you can also build it by yourself. GitHub as well, I believe. I think a part of it.
Not GitHub. I know Azure DevOps in the past, you could have it on-prem. Yeah, yeah, yeah. DevOps Foundation. DevOps Foundation. Yeah, yeah, yeah, exactly. But Jenkins is in the cloud service? Not sure, actually. Why is that? I'm happy to not really use it that often. For me, it's either. But if we are using tools like Defender or SonoCube or whatever to secure your code, is it possible to have that on the other tools as well?
Because in my opinion, Defender is only available at the moment for GitHub and Azure DevOps. So within the Microsoft ecosystem, it's okay. But if we extend that, and we want to extend that to other developers, probably we can make a switch there as well.
Yeah, I think the good thing about GitHub and Azure DevOps and that kind of stuff is that all of these things like vulnerability scanning, secret scanning, and hopefully in the future things like automatic package updates, it's just all integrated into GitHub, into Azure DevOps. For other platforms like GitLab and what we talked about also, you can definitely still
install SonoCube, but it's more of a manual job. But I think, regardless of the platform you're using, you should be using these kind of tools to ensure that what you're going to deploy is secure and meets your expectations. Yeah, and in the modern world, we're using GitHub, for example, or Azure DevOps. And we try to, within the build process, we want to do everything within GitHub, like credentials, like a service connection, like code scanning, everything.
Yeah, definitely use the tools and the options that your provider gives you. Don't start building stuff yourself. There's nothing really wrong with building stuff yourself. Who's going to maintain it? Who's going to maintain that PowerShell script that might be using an old version of the Azure API? Just use Azure DevOps service connections and Microsoft will host it for you. Another thing I might want to bring up into this process thing to kind of move away from
the tooling. So as a developer, if I think about securing the build process, for me it's really important to also make sure that what you are trying to build and trying to publish is secure. There's a lot of articles online about people that accidentally also commit and build their Git folder. So they actually deploy their.git folder to a website. And then if you visit the website, you can just go to slash.git and steal all of the source code.
So that's also definitely a thing that you, when you're going to build something, really inspect what is the result of that build. And am I deploying, for example, stuff like the Git ignore file that's really not necessary? Am I deploying some files that should actually be in the Git ignore? Where is my Git folder in there? Because it really shouldn't be. You want to make sure that your build is optimized. So let's search the internet.
Yeah, there's tools for that. There's tools for that. You can just, you know, it will just go through tons of websites on the internet and just go to slash.git and bam, you got all the source code. You have the whole Git history then from there. Yeah, you have the entire, you have all the source code. And that's also the thing that I always tell people, like sometimes they commit a secret into their repo and they're
like, oh no, I'm just going to put another commit on top of it to revert that. But I'm like, no, if someone has access to your Git history, they can just go back to last month and they'll find the original thing again. The same is for PRs. So some people say, okay, I have multiple commits in my PR and that contains a secret and then, but I merge it and I smash it to one. It's still in the history of your PR, you can find it.
Just a bit of self-promotion. I have a blog post on my web page. No, but exactly. You want to, but if you're using... But how do you control that then? So what you do is you, if you're using stuff like GitHub or if you're using stuff like Azure DevOps, there's this feature called GitHub advanced security or GitHub advanced security for Azure DevOps. And this contains tools that you can just click with one click.
You can enable it and then it will actually like every time you, when you will try to push code to Azure DevOps or to GitHub, it will scan all of your code. And if there's a secret in there, which it just learns machine learning, it's just smart. And then it will just reject your push. And there's it was... That's what GitHub has, right? Yeah. But also Azure DevOps. Yeah. That's pre-commit checking if there's a password.
On the server as well. On the server it checks it. So the server receives a request with the new code and then before it accepts it, it scans it for secrets. And there's also really, really interesting where GitHub does is if it, for example, there's services like SendGrid, right? For email sending. And those API keys, like you said, they are just like permanently valid, which is a big problem. If that leaks, you have a really big issue.
So a lot of companies are now doing that. The API key starts with like a prefix, like for SendGrid, I think it's SG and then a colon. And then what GitHub does is it when it detects a secret, starting with SG and colon, it will automatically send us a request to SendGrid to invalidate that API key. Which is like, even if you leak the secret, it will be immediately invalidated.
You can do that as well for your own projects. You can define your own keys, which it will search in other repositories, I believe. That is useful, but it is scary because like if the secret just doesn't meet that regular expression or something, or doesn't meet the filters, then it will leak through. But yeah, you definitely want to make sure that your code. And also what I do in my pull requests,
I have a pull request template set up. So every time one of my team members has a pull request, I have like a little checklist that you can just create in GitHub and then should develop. So like, did you add documentation? Does the project requirements need to be updated? For example, if you add a connection to a new database, you need to update so people can use that. Also in there, it's a bit of a manual thing, but like, did you leak any secrets? And I
hope to make people think twice. Like first of all, I always tell people, don't put them in your repo and all that kind of stuff. I try to teach them about it, but also make it part of the PR flow that you actually have to visually check that checkbox. It's a check mark that you need to check. And it's visible. So you see it, it's visible and you think twice. That's what you realize with that one. You definitely want to automate it, but it's also just good to teach people. Just don't,
you know, don't rely on tooling too much. Like then people are just not going to think about it anymore. They're going to work in an environment where these tools are not in place and then things are going to go unstuck. In the normal user environment, we have security awareness to educate people with phishing, with clicking on URLs and so on and so on. Within the Defender for DevOps, often in the DevOps, DevSecOps world, we need to create awareness regarding these kinds of stuff.
So do you have a secret stored in your code or do you use other stuff that is malicious, that you can do stuff that you don't want to know that it does? I think awareness is one of the key aspects within a DevSecOps world. Yeah, definitely. That's also the thing that we're trying to integrate, right? Like the DevOps thing, it kind of exists. Lots of companies are doing it. It's so much better than doing
stuff manually or Lisa's computer is the production release machine. You know, that's just that you don't want to work like that. You want an automated system and making it secure is very important. What I really love that I think I mentioned a couple of times on the PR integration of the Defender for DevOps. I mean, it doesn't force the developer to come to a security environment and the phish and phersa. But they talk the same on the same platform that
they need to intend. But if we take a look on the build, right? So on the build side, there are credentials involved. And I think from our perspective, it's often about some ARM templates, PowerShell coding and that kind of, but I think when you are building an application, you have more requirement to that agent where you do the build, right? So bring us to that. What, if you are doing a build of an application, what are the requirements
often that you are meet? Because I can imagine that organizations say, well, you can do this, but it's an, it needs a certain version of Java or net or it needs a certain network connectivity. Do you use like the GitHub runners and Azure DevOps runners or do you use your own stuff? Oh, we can talk about that. That's really interesting. One little thing that I wanted,
we talked kind of about this yesterday and you triggered me about something. So one thing that I want to, I want to talk about really, really, it's really interesting is that there's also now, instead of like building your own CI CD pipeline and having to think about what version of Java you're using, are we using Ubuntu 20 or are we using Ubuntu 18? A lot
of people are still using Ubuntu 18. They shouldn't do that anymore. There's also these approaches in Azure where you can just say, Hey, this is my GitHub link and this is a react project. Go release it. And again, Azure will just figure it out for you. So there's not even, you don't even need to think about it anymore. Just when there's a new commit on main, when some code is finished, Azure will automatically detect it kind of in a
GitOps approach, just push it to the server. And then it's even more secure because you don't even need to think about the builds anymore. Yeah, we have the same with Sentinel for example, where it's a little bit different what they are doing. Sentinel is a Microsoft SIEM solution. So it's where they bring all the alerts and incidents together to create an overview, which we use often for our SOC and that kind of services. But what they do is you have also repository integration. And if you go
and you say click and say, yeah, is it on DevOps or on GitHub? You say it's on GitHub, just as authenticated GitHub. And what they do is they install an extension, which creates a deployment workflow for you on the show. I think even there, there are multiple version because it looks like that Sentinel is pulling that. But because you created that connection, you installed an extension, which is doing... Yeah, it's pushing it. Yeah, yeah, something like that. Yeah, yeah. But that just, it's
just another thing that could be automated and could be secured by Microsoft. But let's go back to what you're like, let's go back a step to what you're saying about like as a developer, what do I need? Because if we're thinking about deploying a bicep or deploying ARM or deploying Terraform, we usually don't need that much, right? I think you, like I'm just saying you guys probably need stuff like, you still need, your only thing you need is probably like some Linux machine with like the...
PowerShell. With PowerShell. Azure PowerShell. I feel like that is all you need. But for like a developer for me, so I'm like deploying maybe an application with a backend that's written in.NET, a front-end written in React. So I'm probably, I'm gonna use modern.NET, so I'm running on Linux. But I definitely need a specific.NET version installed. I need a version of NPM installed, maybe yarn, maybe a lot of project dependencies that it all needs to pull in. And yeah, there's a
lot more dependencies to build. Luckily with stuff like Docker, which we also kind of mentioned this, some of this pain goes away because you can just let Docker do all of this. But yeah, you definitely need more dependencies, which you also hope are secure. So do you use, even when you need to have certain specific version of software tooling, the public hosted, the managed versions of GitHub or Azure DevOps, or do you use your
own? Because there has been, I think definitely in the past, there has been a lot of discussions like sharing and build agent with multiple organization. And there were some scenarios that people could see things that they shouldn't be able to see. Oh yeah, yeah, yeah. I remember it. So yeah, yeah, you're right. I always use that's a
true risk, right? If you are talking about build, seeing, being able. So if you're running your program and I'm running my program on the same build agent and I find a way to escape and then pull your code out there as well, we talked about the connection, right? To take over the connection. But maybe instead of connection, we can step in and change the code or pull some of the code away.
I think it's like, you know, there's this security thing where if you're securing your house, the only thing you need to care about is that your house is more secure than your neighbor because then the robbers will rob your neighbor and not you. And if I'm thinking about like setting up my, either my own like a hosted agent where my code is built on, like I have a VM in Azure somewhere that runs and builds all my code, I have the agent installed
on it or I'm just using the Microsoft agent. I prefer to use the cloud version. We at the company I work in the beginning, we did use the, we just had our own VM. And then, I don't know, there was one guy that always took care of it, right? He was a great guy. He was really good at what he did. And at some point he was working with some other stuff. Four years later we discovered that there were no updates. That's like, and that's the thing. Like if
you're working depends really on the company, on the company culture as well. If there's actually security specialists in there, if there's ops people on there and they, and you're building really important code that really needs to stay in your environment and your people are specialized in keeping that stuff up to date. Yeah. But if you are self hosting staff, then you need to take care of life cycle management.
Exactly. And update, not only updating, but also, also like from a security perspective on day one, we have, I have zero days or you need to update on the second day. There's an open port that is discovered because you need to connect to an agent on a specific port and that's in that specific scenario needed. But two months later, that specific scenario has changed. Yeah. You need to change that port, close it, open another one or whatever,
that you need to, you need to take care of all the stuff on prem. And if you're using a cloud agent, for example, then yeah, Microsoft or someone else is taking care of all that kind of stuff. And you have it on about your own house, secure it more than your neighbor because otherwise you're robbed. But what if you are living in an apartment where you have one house with 10 apartments and the main entrance, the main entrance should be
secured. And that's the people from the apartment building that are securing that entrance. So in that case, it's the same for the cloud agent where I use the same agent as you do with your own company and Microsoft or another vendor needs to take care of the security on top of that agent that is running. But also if it's multi-threat, so I have multiple organizations that are using the same agent and so on. So Microsoft is taking care of
that. Yeah. But the thing is for me that, you know, I'm not a security specialist. I do like to think that I know a bit about security, otherwise I might not be sitting here. But my job is to develop applications. It's to kind of, you know, come up with architecture for applications. My job is not to secure stuff. So I really, I can't, I can't, I cannot in good conscience use a self-hosted agent because I know that Microsoft will do a much
better job securing it. And it's, you know, this is, it's, if you're working at a company, like what is your core thing that you're working on? Like a web shops core thing is building a web shop. So should they also be building their own identity provider? No, they should use a built one. Are you, are you in a very, are you working in like, I don't know, like an airline or like at a bank, you might be using your own agents because it's really
important to you that you need to manage it yourself for normal companies. This is just not their thing. So I really advocate for using the cloud version and Microsoft has people working there 24 seven to keep that stuff secure and any zero days should be fixed immediately and you don't even need to think about it.
From a life cycle perspective, I totally agree. But if you, let's, let's look in that apartment house where Microsoft needs to close that, that, that the first door, if you are running multi-threat agents and you are running on the same agent as I do, how do I, how do I know that is secured, that I can access your code in case of, I think, I think that's more of an implementation detail, if I'm honest. Like I know that there's probably like, let's
say there's 10 VMs, right? And you're thinking that you're working on the same VM. It's probably some Docker host. It's probably some hyper V stuff. Maybe it's running on the same machine, but the only thing you may be sharing is a kernel underwater. I'm not sure even how it works. I don't need to think about it. Well, I think it's, it's, they are changing. Definitely with GitHub, um, it hasn't been a big shift on Azure DevOps. It was mainly on, uh, I hear myself very weird. Is it me
or no, no, no. Okay. Um, on Azure DevOps, it was always a hosted windows and the agent just run that. Um, but when we went to the migration to GitHub, GitHub's all, uh, hosted, uh, uh, containers. Um, and I believe Azure DevOps is also now using mainly because they share that backend. I mean, I've been using hosted Linux agents on Azure DevOps for at least six years, I think. But are there now really VMs or are there already changed to a container? I believe.
I'm pretty sure you can run a Docker container with the Azure DevOps agent in it. You can definitely do that. So that would already make it easier, right? You can just kind of get the latest version of that agent and it will update all of the vulnerabilities. Again, something you have to manage yourself.
Yeah. But if I want to deploy that to an Azure subscription or I deploy that, that specific application on a certain environment, do I need to integrate that agent within my own environment or is what are you? That's a good question. I mean, one of the, where do you actually, where do you put it in your infrastructure somewhere? Where do you put it in your landscape? One of the most, um, like most often attacked environments, I think, or one of the most popular things
to attack in an, in a company is the build server. Cause sometimes people don't really care about it. They don't keep it up to date. So if you can just hijack that, you can just find all the source code there. You can find all the credentials there. So maybe you want to sandbox it, I guess. I feel like you really do not want it to have access to all of your own premise or all the other servers. At the other hand, if you're deploying to virtual
machines from the build agent, it has to have access to the VMs, right? Cause it needs to be able to come to put the artifacts on there. Yeah. So that's, I think, um, in my opinion, one of the biggest re, uh, wall, one of the reasons why we see, uh, that people are, or customers ask for having hosted self hosted agents. It's often, it has to do with networking because they have like networks, databases or something in the on-prem environment or in a cloud environment, which is not the same
is you will get the same issue when, for example, if you go into cloud, right? If you go to, let's say you have Azure web app, uh, where you have everything, Fina integrated and you have Azure SQL as a backend. Um, so your Azure SQL is not publicly accessible anymore. And then you are going to do a build, uh, you are going to push the changes to your, uh, SQL database. That's not accessible anymore. True. I had a little note about that because some time ago I read that GitHub now does
support this. So GitHub supports using a cloud hosted agent. So you don't need to think about it anymore, but you can then talk to V net integrated stuff on Azure. I'm like 99% sure I'm sure I'm right that I read about this. And then of course the first million comments were when will this be released for Azure devils? So I just looked that up because I believe, I believed it was completely impossible to use like a Microsoft cloud hosted agent
and deployed to V net in Azure. And it kind of is the thing that Microsoft says on their documentation is that you just need to whitelist some IPs from that. And they change every week. They say they could change every week. So every week they upload a new list and then you need to upload those and whitelist those every week. But you can automate that. Yeah. But that's not new. That's the same as whitelisting the whole Western Europe data center IP address
space. I mean, I had once a customer that said, yeah, give me the, I'm not going to whitelist that. Give me, I'm not going to open a firewall. Give me the IP address that I need to whitelist. You can download it. It's thousands and thousands of IP addresses and the changes that they remove. They remove IP addresses and so on. So changes are. So one of the, one of the other reasons we use ourselves, we use a self hosted containers
on Kubernetes. And one of the reasons why we did that is because through that model, we have now the option to use the manage identity. You can do the same as well with the self hosted. So in Azure DevOps, you have like two versions of VM agent, but the one that's with the manage identity, I believe, I don't know. You can create like an Azure service connection with a managed identity and that will be used by the cloud hosted agent. Yeah.
So you, what you do is you use the manage identity of the agent to authenticate to your Azure subscription, for example, to do some activities. Yes, exactly. So that's also a really awesome. Well, that's kind of the same thing you're doing, right? You just have a container. Yeah. You have the kind of the same thing. I want to ask you a question about
this. So let's say that we need to, let's think in a modern way. So GitHub, if I'm correct, has the option to use a cloud hosted agent, which you don't need to update, which you don't need to maintain Microsoft or GitHub cares about all the security aspects of it, which is great. I can sleep at night. And let's say that it's correct that they can
just talk to a V net integrated resources. So things that the internet cannot access, let's say Azure DevOps gets this as well, which you then still want to use self hosted agents. So for us it's double. Um, we do it partially for, so we use the build agents partially to use the manage identity, for example, to configure and maintain our own solutions,
right? So our own platform and that kind of stuff, which are not internet facing. Um, but we also use that manage identity to do other stuff, uh, other resources, for example, without going too deep in it. So, um, how do I say it without saying too much about this? That's the fun part. But what you can do is for example, if you're in a scenario that you are, you're an MSP, right? You're not doing nothing with security. You're an
MSP and you are using Azure Lighthouse to manage your customers. Normally you need to have a service principle which is going there. So you have always an epi registration with a secret, which is vulnerable for that. So in that scenarios, you can also decide to go to manage identities of your containerized solution. So when you are doing something as infrastructure scope to your customer or you are doing as your lending zone configuration,
you can use those connections, right? Um, to do the proficienting of that. So in our opinion, being able to using that manager identity, um, but it solves a lot of the problem that you initially mentioned. I don't really understand the difference though between using a, like I configure like a managed identity in Azure for my Azure DevOps or GitHub action build, right? Build agents. The one that's made by Microsoft in the cloud somewhere. Um, and I only use that one for pro deployment
permissions. It's not going to be able, it's not going to be allowed to like, I don't know, change Azure settings, you know, that as little access as possible. You're doing kind of the same thing. You're hosting your, uh, build agent yourself, but you're also talking about the managed identity access. But how does that differ from what I'm saying with the service connection and managed identity? Um, service connection with the managed identity.
Yeah. You're saying like on your Kubernetes environment, you can like configure, uh, it can be a VM as well. It can be a Docker, uh, container. Yeah. Your pod in Kubernetes has a managed identity associated with it. And in my Azure DevOps cloud hosted agent also has a managed identity because of the service connection. Yeah. But it's on the VM, right? On the cloud hosted. Yeah. So, okay. So is your, your, your, you don't have on the cloud hosted one. You don't have managed identity, right?
No, no, no. But the cloud hosted one uses a service connection to talk to Azure. Yeah. And that service connection uses a managed identity. That is not possible. All right. Pretty sure it is a federated connection. I mean, I'm pretty sure it is possible. Okay. Let's just say service principle because the idea behind managed identity is that the identity is not managed by you. Exactly. So you don't know the secret of it. Exactly. And I, I, um, maybe we need to take this offline podcast, have
a look around. Maybe we can put it in the show notes or something like that. Something like that. That would be awesome. In any case, what the thing that, um, let's say that the security aspect would also be managed by Microsoft. Yeah. So the, the, the update parts would be managed by Microsoft. I could integrate with V net. I could integrate with my private
resources. Would there still be benefits to using self hosted agents? If the security aspect was handled by Microsoft in the cloud and if the V net integration was handled by Microsoft. Um, well, um, depends on your level of security, I think. And the, the application and how much risk you accept. For example, if you are doing some infrastructure code and deploying some resources, it's way different than if you are building databases and application
and which are doing financial stuff or true. Yeah. That's a good point. So I think it's an, there's always a risk with the public hosted by someone else. Um, from a modern way I do accept it. Like if you're a small organization, you don't have the resources, don't start with it. Right. I do completely because you need to manage it. You have lifecycle management, you need to patch it, you need to monitor it. You need to have the fender
on it. You need to alert, react on it. So yeah, on that sense, I do agree with you guys that that if you don't have the resources and the capacity to do it well, um, then don't do it. But if you have like, but I think that's in always the case. If you have the resources, the budgets and to make, if it makes fun, if it makes sense financially, if there's like, if you did like your risk assessment and you're saying, well, this is a pretty
big risk for organization. It depends on the user. It depends. That's a great consultant answer here. Definitely. I have one last thing. Um, we talked about to build process, the built environment, but if we want to access from that development, a developer perspective, if I want to access that, uh, that the built environment, the, to, to create code and so on, you're doing that on your own laptop. I'm creating code on my own laptop is what
you're saying, right? Yeah. I'm using my local digital studio or rider, stuff like that. How can, how can, and maybe we, we, we need to take another episode on that, but how does dev box within Azure Azure dev box? How does that come into place regarding the things that we're talking about? Because securing your build, uh, stuff, right. It's also securing the entrance, how you access your build. Yeah. I think that's, I think, I think we talked
a bit about this in the first episode about workspace and management a bit. Yeah. I talked to a colleague yesterday cause they were working for a client and I was like, so what kind of environment are you using? He said, like, I'm using, I'm using the, uh, the Microsoft,
what was it called? Microsoft box in Microsoft or Azure dev box or something. And he said that works really, really well, which we talked in the first episode about it, that I've heard of people were using it or like stuff like GitHub code spaces, which is even more like as a sauce platform. I think it is definitely very interesting. And we could maybe dedicate another episode to this. Maybe I miss it. And maybe what that also our listeners, can
you maybe, uh, what is that? That's a great point. Yeah. Let's talk about that. The, the dev box is, um, more or less a workplace within Azure that you can spin up a sort of an ACD environment. Um, it's not AVD. Don't don't go to virtual desktop. Yeah. It's, it's not an AVD environment, but it's a virtual workplace that you run in Azure. So you can connect it remotely, but it's dedicated to a subscription or a, a, a, an environment that is used to
develop stuff. All the, all the software needed for program software development is already installed on it. So instead of like getting a new laptop from your work and spending two days configuring visual studio, official leader code, GitHub, get crack. It's already all done for you. So you can just, you know, click on start or there's a new developer. It's a VM still it's under the hood. It's a VM AVD probably as well. It's just AVD on steroids,
I guess. Yeah. It's not AVD, but it's, it's a VM where you can, yeah. But otherwise, I hear people are complaining dedicated decision about definitely, but for interesting about the dev box and that's, that's one of the nicest experiences that it's dedicated to a subscription. And with that, from a cost perspective, that subscription belongs to one project. So you have a project, you're running that project and what are the costs
of that for the prep, the running costs for that project. So you have a subscription, all the costs are on that subscription. So you have a one project with that, with that costs. So it helps also when you hire externally, external people, they can get access to the dev box and exactly. And only if you are running a bill pipeline or whatever, it's only based on that specific Azure subscription or multiple Azure subscription that you are attached to
the dev box. You can access other resources in your environment. So it's really specific. But I think it's definitely, I definitely have some more questions or things to say about this. I think it's definitely good to talk more about this later, but one little question about this. So how the build process, how is this related to dev box?
Now if we, the thought that was in my head was if we are building a, an application and so you're running a pipeline or whatever, we can secure that on an optimal way, but accessing that bill pipeline, if you don't secure the access to your bill pipeline. And we talked a little bit about in the first episode regarding a workplace, right? But on the other hand, you can access a GitHub environment, you do authentication and multi-factor
and so on. But if you have your source code, source code within a official study, your local machine. Yeah. So yeah, exactly. That was the reason why it pops up in my head. Exactly. You could, you could definitely secure and like put all the security measures in place to make sure that your code is deployed securely onto your, well, the thing you're your, your landing zone, wherever you're putting it. But yeah, then we kind of go back to the
question of how do you secure the access to the thing that is being built. And that could be exactly something like dev box, like where at the other hand, I feel like you could, you know, pry this laptop from my cold dead hands. Like don't, I love having a laptop, but that you're completely right. If I left the company somewhere, I could still have
that code on my laptop. I could take that with me with dev box. Yeah. You could just, you know, delete, disable my Azure account and you don't have, you don't need to format my computer anymore. It doesn't need to collect dust for five years in a closet somewhere. I think I know someone from, from the Microsoft team who is on, on that box. So we probably can invite a special guest to have a specific scenario or a specific episode on that box.
Let's do that. Sure. It's really good. And regarding the build process environment and stuff that we do, do we need to take another thing or did we miss something? Maybe, maybe a simple thing that I sometimes see people do. I think we, we definitely talked a lot about making sure that your build process is secure, making sure that you have like a well fitting build environment, like either self hosted or like cloud hosted, making sure
that it fits. One little thing that I sometimes see people do is that they're kind of new to this build process and they're like, Oh, I need a specific task. Right. And then they're looking on the marketplace for like GitHub actions or Azure DevOps tasks. You really want to make sure that the tasks you're using are okay tasks. Like there's even settings
in Azure DevOps, I believe to not allow marketplace tasks. If you could have everything secured, but then you download a task that just copies all of the source code and sends it to Russia. You could, so you definitely want to make sure that the tasks you're using are secure. They are, they're trusted. And yeah, cause you don't know, you don't know the source code of the task. You don't know what they're doing. So the things that are happening in
your build, you want to make sure that those are okay. Those are secure. That's a really good one because I think that's happening more often than people. Yeah, exactly. I need to replace some tokens. Oh, and I know that this one has a 600 downloads and the last update was four years ago, but it does what it needs to do. So I'm just going to use that
one. Like, ah, so we spoke about the technology and I think the last one was great, but is there also a human interaction that we can discuss if it comes to our bark, more role based access requirements? Other, what do we expect from, um, our team when it comes to, I think if you're like running a build pipeline, it's also definitely, you want to make sure that it's only specific people can run specific pipelines where like deploying
your cloud resources in a really big corporate environment. Maybe only the, you know, the, the ops people need to be able to deploy actual Azure production resources. There needs to be the four I principle where if I want to change the connectivity, uh, stuff in Azure, you know, all of the V net stuff, all the connections of Azure firewall, you want to make sure that your colleague Steve, that is also really good and you know, works with
the, is specialized in this. He also needs to have a look at it. That's kind of, I guess the human aspect. You want to automate your, automate your build process. That's the entire thing about DevOps and kind of, but you want to also make sure that when specific and important things are happening, that's, I think where you need a human to double check.
Yeah. But also a role based, you have a pull request. It need to be approved and so on just before the belt is actually running for ice principle, maybe six eyes that you want to cover. I think that's, that's important. And also what you mentioned a few minutes earlier regarding the, the, the, the list that you need to approve the check mark that you need to do before moving into PR. I think that are things that awareness we talked a
little bit about that, the awareness also within the DevSecOps, the DevOps people. Yeah. That needs to be, that, that needs to be raised in my opinion. Yeah. We want to make sure we want to know what we're deploying. Like the, I think that's, I think that's a good way to say we want to know what we're deploying and we, the deployment
should have an automatically, that is what we're doing all of this for. We just want to have code and we want it to run on production and we don't want to use Lisa's computer to do that. We want to use a cloud hosted agent, a self hosted agent, just let it happen. But what is being deployed? So you don't know that thing. It's like a whole, it's a whole old joke about, you know, I don't know. I don't know. Introduce her to me. No, but I
don't, it's on the flight and the eight. Sure thing. So like one specific, one team members laptop is always used to like, deploy to production and that's an old man, whatever we all know that scenario with customers that there's all always one server that has enough permission to do the deployment. Exactly. I have faced that personally with a, with a really big enterprise customer when it came, every need, everything needed to be dev ops
and everything needed to be infrastructure scope. But then you needed to email the code to somebody and then that person went to an SSH to an Ubuntu server because that server had the right login already to the Azure site. So they run it manually then and they, oh, nice. I can add one more thing to add one more thing on top of that. I used to do an internship and I also like finished my studies at the Dutch tax services to the blessing
needs. And it was also, I worked at like a very modern department there where they built a mobile app. So it was really fun. And they also started doing like deploy and build agency ICD in there. So, you know, the code was built and quality was tested, like all the stuff we talked about today. And then there was a step to go to a specific Bali, you know, to go to a specific place in the company where you had to fill in an A4 paper. And then afterwards
the process would continue, which I just found really, really funny. It's all super automated until you get to the deeper and deeper parts where you actually need to write a piece of paper and go somewhere. And yeah, it's not, it's not, it's not particularly wrong. It's, it's good that you have to sign off, I guess in a government environment. Yeah. Yeah. On the other hand, we want to try to digitalize everything. And then if you need to sign a
paper, it could be good. It could be weird for some people, but on the other hand, if you have, you need to take that step. So you edit for awareness perspective, I think is really good. Yeah. Okay. I guess. Yeah. I mean, everything could of course be optimized better, but you do definitely have a point, especially that's more to cover the change management than I think. Right. I guess a little bit like who is, who is authorizing
the deployment, you know, that kind of stuff. Yeah. Because that's fun to me. Big important thing, I think in our next step, because I believe the next one is going to be about release then. I'm not really sure. We'll have to, we'll have to take a look. Yeah. Okay. But because the change management is going to be a big one, I think, because when it
comes to build and deployment, we deploy more often than we did previously. Right. So maybe in the past we did collect all the changes and then we did a deployment once a month or something. But now we are doing multiple ones. So, um, and this is a really, uh, challenge for this with a couple of customers of mine that say, okay, we need to be in control. We need to see how much deployments we need to the production environment. Um, but I think
we'll take that for the next episode. Let's close this one. Yeah. We are almost an hour or so. We make that. Um, let, let's look a little bit back. Um, we covered the build pro the build process, the built environments. Um, we talked about a little bit dev box, make a little bit, a little step outside. Um, we talked about securing code, uh, also within this episode. So, uh, we, I think there are lots of aspects within the build that
comes into place. Um, and that's make the mix. This one of the interesting topics, the steps within creating, uh, an application or building, um, an infrastructure, uh, you need to take care of your build because a lot can go wrong within this step. Yeah, definitely. So we talked about it, about the defender for DevOps, um, in a few episodes earlier
on the, uh, what was it? The defender for service, uh, of the, uh, and for series. So if you want to have, you are interested in that one, uh, take a, have a look at that episode. Um, on the other end, that was it for today built and that let's look into the next one and probably release or something in that, uh, maybe at summit we'll see. Yeah, let's see if we can connect with, uh, with, with some people over there and see if we
can create, uh, some other content, uh, on the other side of the ocean. Definitely. Let's do that. So thanks for today. If you liked this, uh, this podcast series, um, like and subscribe, um, and also, um, let us know if you want to know more about a specific scenario or what else, uh, we will take care of that and see if we can create, um, an, an next episode on that. So reach out to us on the socials and, uh, I see you next time. Thank you guys. Thank you. Thank you.