Episode 31 - Container Security with Josh Duffney

In this podcast , we will dive into the world of container security , but before we get started , I want to remind you to follow us on social media so you never miss an episode , and it will help us reach more amazing people like yourself . Today we have a special guest called Josh Duffy . Could you please introduce yourself , please ?

So I wanted to be a video game designer because I played a lot of video games in high school , decided I didn't want to have $100,000 of student loan debt right starting life .

So I decided to go to a community college and the only thing computer related they had was this computer networking degree , an associate's degree , and it turned out to be kind of an introduction to everything . I had a networking class , a full semester on printers , which was very painful I still dislike printers to this day .

But anyway I landed a help desk job from there , got really into like system administration , worked my way up to like a one man IT shop running a small bank and racking servers , and then I kind of got into PowerShell and so a lot of people have seen my name before . It's probably because of the stuff that I used to do with PowerShell .

But anyway , that led me kind of on the DevOps path . Previous to Microsoft I was an SRE at Stack Overflow and then I got recruited in by Mike Robbins , who's familiar in the PowerShell community too , to be a docs writer or a content developer , I think is the technical term for that , and I focused on Ansible and Terraform .

I had written a book on Ansible and successfully implemented it in a couple enterprises and stuff previously .

I did the docs writing for about 18 months and then I got the opportunity to join the cloud native team here at Microsoft in the DevRel organization for advocacy work , which was a good split for me between some of the content creation and speaking which I enjoy doing , and I'd done that a lot in my past .

I ran a PowerShell user group and other user groups before , spoken at many things , different conferences , so it seemed like a good fit . And then I landed somehow in container security for probably like the past two years or so . So that's like I don't know how long have I been working Like 12 years , I don't know . 12-year snapshot right there .

So I'd been working in the cloud for a number of years , probably before the term cloud native came about and just I don't know , it seemed like the things that I was naturally doing in the cloud and then having them change , I suppose , from on-prem and stuff , and then it just became a term .

So it's one of those situations where you've just been doing a thing for a while and then all of a sudden , uh , the broader ecosystem comes up with the term for what it is that you've been doing for a while .

Back in 2016 , I did a talk where I wrote a module that abstracted DSC into JSON , so that's pretty cool to see that . I don't think they used any of my work or anything like that , but just that idea transferred . Yeah .

I've been working with a couple open source projects Notation , which is a digital signer you can think of it that way yeah , and then they're ratify , which is an emission controller for Kubernetes , and then copacetic , so that would be for the patching , and all those are related to the software supply chain and containers .

So container security most people would think immediately think chain guard that's really focused on the images and make sure that those images in the best possible way . That's a big part of it . A lot of the work that I've been doing is on somewhat on the other side . Right , okay , we have this good image that we trust or want to trust .

How do we signal that trust ? And that's where the signing comes into play . And then how do we know what's in here and that would be something like an S-bomb and all of these are . It's been an interesting thing to watch in the industry because it's more work and definitely not something that the average developer wants is more work .

But there is a lot of pressure from especially like government agencies and working contracts with government agencies where they're now mandating these types of things to improve the supply chain , to avoid supply chain attacks from happening . It's an evolving ecosystem .

epiphany as I was learning Kubernetes and getting into containers , when Docker was really starting to come on the scene . And one of my first thoughts I remember because I had battled Windows updates like my entire career VM updates and I was like , oh , finally I'm not going to have to worry about this problem again . Here we are , 10 years later .

I'm having to worry about , you know , vulnerabilities of my images that I have to patch or you can patch by updating the base image . We can get into more detail with that later , but at the end of the day there's still vulnerabilities in these container images and systems . It didn't solve the patching problems per se Exactly .

You know like we still have to have systems around updating things to make them not vulnerable .

So there's another tool I haven't actively contributed to , but I use all the time , which is Trivy . So Trivy is a vulnerability scanner . It's probably you , might it might be using your tool , might be using it .

So what you can do is you can hook these all up with GitHub Actions , which I'll show you in a second , and then the next step would be wouldn't it be super cool if we could remediate those vulnerabilities based on the report ? And that's where Copasetic comes in .

So Copasetic will take care of OS level vulnerability patching for container images and it can ingest trivia results to do targeted patching . It also has the ability to just patch everything to the latest on your system If you want to .

You're feeling risky and you want to , you want to go that route , you can , but the trivia one will just I found a Cve for libc , I'm going to update libc or whatever that might be , and the report takes care of that , uh , and then ? So now we have this image that we feel pretty good about . How do we signal trust ?

And that's where the notary project or the notation clr comes in , and you can more or less stamp . You put a digital signature on that container image , a notation , so another another option would be Cosign from Sigstore , so you could use that as well and that creates a digital signature on the image and that becomes important later on .

But there's a distinction between . Well , I think they can do both . If you use the beta features for Cosign , they both can use the referrers API or it'll create , like another artifact , a signature artifact on your registry , and I think I even might have some packages that we could see what that looks like on here .

We'll see if I do no , but maybe yosh does so .

tenants and stuff , and that's just the nature of my role in the advocacy . You know I'm not responsible for any internal production systems . I don't want the Wasm cloud , it's all it's my cloud .

Is this the right one ? Now I want packages and not there , so so this is a signature here . This SHA-SIG , I guess . Yep , and so that's what Cosign does by default is it'll put the signature on the repository Notation by default , will make it through the refer , so it won't show up here .

It'll actually be in , like the manifest metadata , if you were to like pull up latest and in here we would have some information to the digest for the signature and it can do that with . It works with , I think , most registries , any OCI compliant registry it works with .

So we'll pause right there , though , and just you know , like why would you even consider patching container images when , like , this is the build pipeline ? I could just bump up , uh , the base image version . When you can do that , then you know , I would say patching , don't do patching . I think what was it ?

Chain guard just had like something on their x profile of a big billboard that they put up , that they had like patching something , and they crossed it out and they said secure future . I don't disagree . Like , if you have the ability to use a better base image that doesn't have CVEs , do that .

The problem comes when those vulnerabilities are upstream and you have a dependency on some other image and their cycle is longer than yours and they can't resolve that cve . You're basically locked right , like I can't upgrade a version because it's going to break something or because there just is another latest version yeah , not a lot of the like couple seconds .

Technically , it's super cool using build kit to to break apart your container and patch it and create a new page . It's pretty cool .

Like there would be another tag for a point 1.25 , you would have another one that is always patched . That's , that's one tagging strategy that you could you could do . That'd be the dynamic one . The other one is the , the incremental one .

So you can just see there in the docs , you know , like the first time it goes through and patches it's going to be hyphen one . The second time will be hyphen two . The problem with that is you , you have to go up and up , you have to go back and update your manifests and I'll have this ready . This would be cool to add in .

But I'm working on setting up a sample repo that uses Dependabot to be able to come back and say , hey , I detected a new patch version of your image , just like it would any other dependency update right , and do a PR into the repo to say , hey , let's update the image to a new patched tag .

The issue with the other supply chain tools down the line is if you're not using the tag and you're using the digest . So , for context and people that don't know , each tag has its own digest . If you look back at this , the SHA right here is the digest , so you can actually reference container images in your manifest by digest .

And that's the reason that's good is because it's not mutable , meaning if the image has changed in any way , it gets a new digest and it won't pick it up , which is inconvenient if you want that . But it also improves the security , because now you know for certain which exact tag and version of the image you are going to have deployed .

But if copacetic patches changes the tag and then that digest is actually used by the signing tools . So the other tool that I didn't get Okay .

enterprise you have your own keys . Notation is what you want to use . But if you don't and you just want to have a signature , cosign actually has something called keyless signing where it will basically get rid of the private key so it can't be used again , but then the public keys are made available . I believe it was a cold record .

So that way you can verify the signature but you don't have to maintain your own PKI infrastructure . So that would be the big delineation , at least in my mind . If you've got a mature PKI system and you need to use your own certificates , that's where notation shines .

And then cosign is on the other side , where maybe you're doing more local development or you just don't have the infrastructure , don't care to have the infrastructure and you want signing , and I'm sure they have offerings to kind of offload the PKI infrastructure and they have , I'm sure they have offerings to kind of offload the PayCat infrastructure .

built in like monitoring on it if we just update the image accordingly using automation . Whether it's GitHub or Azure DevOps Yep , there's something that can be built in like monitoring on it .

Yeah , a product could definitely sit on top and use all of these things together and make it a lot more seamless , and I think that's honestly what needs to happen to make it more palatable for companies and organizations . This is a lot of additional work , but this is what it would be doing under the hood . There are a number of features inside Azure .

One would be the continuous patching for ACR , so it uses ACR tasks to do the copacetic stuff for you and then it takes it out of your workflow and makes it more of a platform checkbox type task .

You would see your vulnerability list . It's not displayed on any kind of dashboard . It's displayed here . There are flags with trivia that you can use to stop your workflow and fail it if a certain number of CVEs exist , even after you patch or during the scan .

Do you know any examples where this is used in companies , or it's just microsoft at the moment ? You know where ?

So , if we look at the bottom part , so it assumed like we've got a signed image on our registry and now we want to deploy to our cluster . How do we put some teeth to all this work that we did during the build step right To produce this artifact that we trust ?

So ratifies and gatekeeper combined , so you deploy them together with a helm chart , and what that allows you to do is do admission control for digital signatures or other verification policies , so you can define some kind of rego policy . And in this example it's looking for does a digital signature exist for this public certificate , right ?

And so if I can't validate the signature with my public key , that signature either doesn't exist or is invalid , and I'm not going to let the pod run on this cluster . And so there's a lot of good use cases there , trying to think what are the other customers ? By name or if I should say names , but yeah , there are definitely customers using it .

A lot of these are actually being baked into features right Like features inside ACR features inside AKS or whatever it might be . For example , like with Ratify and Azure , you can define an Azure policy that converts to a Ratify configuration , so that way you don't have to apply another kubectl thing .

It can just be any AKS cluster that gets deployed to this resource group has to

have signed images , and then that translates down to Ratify .

I'll show you a more recent one for people that are more on the platform engineering side , because a lot of this is being done on the developer side , which might be hard , and maybe you just care about patching the images in your registry and I'll talk about a continuous patching workflow here in a second for at least GHCR .

So yeah , if we just let me bump this up , the general workflow is we're going to log into Azure , log into ACR , we're going to build our app , right , and then we scan it . So we scan with the trivia action that's going to output this patchjson and then we're going to patch it . So this can be replaced . There's a GitHub action for this .

Now this can be replaced . I'll show you that in the next one . We're going to patch the image . So I'm just outputting what the digest is going to be and then push the patch image . So I'm going to push the patched image after Copasetic patches it . So Copasetic is going to patch it .

It's going to create a new image locally with the taking prefix like hyphen one or whatever taking strategy you're going to use , and then I'm capturing that digest at the most optimal time , which is right after it's built . I'm going to capture it .

So there's no chance that I'm I'm grabbing the wrong digest and someone put another layer on this image with something malicious in there and I didn't know . So I'm going to grab that digest right there and that's where notation comes in .

Notation has a , an action as well , so I'm just setting it up and then I'm signing it and then that puts a signature on the refer API that can be validated with ratified later . And then so if you looked at , I think I have terraform code in here . This sets up the aks cluster and then I'm using it in the demo I actually set up ratify .

So it's not , it's not part of my infrastructure as code , because it's part of the demo script , but all this you can walk through as a demo . This demo sh goes through every single step in the workshop , so that's from like a pipeline perspective . But let's say that you wanted to like what would it look like to continuously patch a registry ?

And so I started this repo and I walked through this in a blog post series for people that are interested .

I wrote a little go CLI . I turned it into a CLI instead of just a go file , which we can see in another branch . There's actually quite a bit of logic in there to detect the tags . So the advanced logic that I put in there would basically calculating the tagging strategy . So I'm only going to grab the latest .

If , like , net monitor has a hyphen one , I'm only going to patch hyphen one and not zero , because zero came before hyphen one , and then if I patch both , they'd be at equal patch level . You should , you know . Then why not just flatten it down to one tag ? So there's a little bit of strategy there .

And then I'm calculating the incremental on that and that looks like let me open a new tab . The output for that CLI improved looks like this . I called it contiguous . I thought it was clever . Maybe it does .

I don't have a screenshot maybe , but essentially you would do this list and then it would output a JSON of here's your current tag and here's your next tag . And then GitHub Actions is pretty cool because you can just use from JSON to generate your matrix , yeah , and then basically go through .

So for each image in the matrix that I identify that needs a patch , I'm going to go through and generate a trivia report . I'm going to check the vulnerability count . So if the vulnerability counts zero , I'm not going to bother patching , because Copasetic is just going to fail and say I couldn't patch anything .

This patch tag is taken care of and then the new version is CLI . But this is just some bash Again . This is why I've had to learn more . Bash is because of GitHub Actions , to be honest . This calculates the next patch tag based on a tag , then it goes through and runs Copasetic logs into GHCR and then pushes the patch tag .

So the next part of this workflow would be like okay , I've got all these new patched images on my workflow . How do I automate that final step , which would be the PRs , into the code repos that are using those images ?

This was just to give an idea of , like what you could build with it and a lot of people are doing with that . I I think , yeah , in

ACR they're working on the continuous patching thing , solving a lot of these problems as you dig into it . There's issues , like I mentioned , with the signing , like how do we make sure that the signatures are still valid ? How do we update ? It's funny just last year I contributed the logic for Ratify to support inversions of keys .

So basically it'll look at your , your keyball instance and if you have more versions that you've created for a key , it'll update , auto update the cache , which solves the next problem that the continuous patching does like . Okay , what if I signed it with a new key ? Do I have to go then update my verification engine ? The answer is no , it'll .

It'll dynamically pull um . So that was kind of cool to see that that work get used , okay so , yeah , it looks like it's quite a good tool to quite batches .

The only thing that you'd have to figure out if you're not using GHCR is figuring out what tags you want in your matrix , and you could manually define that . So if we go back and look at this one , I had an example . I had an example of using just a static matrix .

So if you knew the tag names that you wanted , you could just define your matrix like this , right ? But I wanted to be a little bit more pragmatic , and so then that required me to go in write my own logic against GHCR . So that would be . The biggest gap right now is you need some mechanism to feed into the copacetic workflow what images you care about .

So if you want , you can use my Go code . That's fine .

But if you want to build your own type of solution , all the tools are there to improve the supply chain and again in the CNCF , so like they're stable and you know that they've got good investment in them , they're being used internally for Microsoft . They're being used internally at other big competitors too .

One interesting thing AWS and Microsoft are both working on Notation , so they're collaborating . We're collaborating on that one . But yeah , so the tools are there to put it in there if you want to go the open source route . But I'd imagine there's going to be some cool tools that kind of wrap all this and make cool button Right .

Your spare time , josh , like your job as an advocate on your job and then learning about security in containers .

Lately it's been I've actually been revisiting some old books PowerShell books , no , actually more like productivity-type books Deep Work by Cal Newport , I think it was released in 2016 , 2018 .

, I'll burn through the sample on my kindle , uh . But I do prefer a physical book , um , even technical books , which are more annoying to do because you have to . But anyway , yeah , I like the physical book . So I I don't know , I'm , I'm all kind of over the place . I read a little bit of Rust , worked with a colleague on some Rust recently .

At first they just look like hieroglyphics , but they're actually very useful shorthand macros that Go doesn't have . That makes it made the transition a little hard for me , but I do really enjoy Go still . It's made me learn things at a much deeper level , that's for sure .