Omer ready? Okay, I think we're live, action. Action! Alright. Okay, hello everyone and welcome to our 8th session of DevOps Topics. And today we are going to talk about application security. Right Omer, security? Security, yes, application security. Exactly, application security. Okay, so Omer as I always ask, I'm glad that you're here, what's the first thing that comes up in your mind when I tell you application security? What do you want to talk about when I talk about this subject?
It's a big one and there are lots of stuff that comes to mind. The first one is the famous list of OWASP. OWASP is the web application security, something, something organization. And they have a list of top 10 that I think updates every few months or every year. It's worth following and taking a look at. There are things that I think everyone know like SQL injection, insecure design, flaws in access control.
Every typical thing that I think most web developers or most, excuse me, application developers should be familiar with and it's worth checking out. So we can dive into one of them, into many of them, but it's worth checking and another thing that I think is really interesting is really into a couple years back is the world of bug bounty. Do you know that? No, what is that?
So there's a whole world and an industry and actually an economy of people that sit at home and hack on companies, legally hack on companies that are participating in a, what is it called? I think a bug bounty program. So you can either, like Google, have your own bug bounty program, which means you can say to the world, hello guys or hackers or whatever. You can hack on this area. This is a CIDR block you can try and access. These are APIs that we want you to try and scan.
Do it in a, let's call it a standard fashion. Don't try to DDoS us or break anything, but do try to hack it. And if you find anything, we'll give you money for whatever you find. If it's a huge flaw in the system, you can get tens of thousands of dollars. If it's small stuff, they have like the list of prices. I think Facebook, the new meta are doing something similar. If you're a small one, you can take part in a bug bounty platform like hacker one and what was the other one, bug crowd.
Then there's an integrity, I think is the third one. There are a few of them and anyone can take part of them. And you just let people scan. You can do it certain areas of your product or the entirety of it. And when people find stuff, you can pay them for the flaws they find. So you kind of get security as a service in that shape. Okay, cool. So we'll put links to all of the things that you just said in the description, right? Because we are nice.
And I'll start asking you questions because application security, as you said, is a huge topic. So let's talk about specific stuff because we like to be specific. And again, we'll try to be focused on the why. Okay, we love to talk about the how because I think it's easier, but we'll also try to dive in and talk about the why. Why would we do this? Why would we do that? Okay, the logic.
So the first question is that comes up to my mind is when you're talking about security, let's talk about containers because we love all the container world, right? All the Kubernetes and buzzwords and stuff. Okay. So when you say application security and you talk about containers, now let's also divide that because we can have security for a static website that is hosted, for example, in S3, right? Right. As a simple storage.
You're usually not much there because it's static and there isn't really much going on. But yeah. And it's more of securing, let's say, the client side, like CSRF, as you said. You know what? One cool thing about what you said. Regardless, whether it's static or not, one of the biggest areas you can find, it's not bugs, but really find anything is the JavaScript part of websites.
If you just go today to a random website, try to find something that isn't that big, like a ministry, the foreign ministry of your country or something like that. Open the JavaScript five. In many, many times, you're going to find very weird stuff like comments that developers left and keys and secrets and maybe things that they don't even think that are security vulnerability, but you find amazing things. So JavaScript fives and there are scanners on the Internet for that.
I can leave a few in the description. There are scanners that will just go through. You can give them a list of URLs or sub URLs and they just scan everything. They will scan the JavaScript files and they will try to find, it's called entropy of strings and it's based on different configurations you've set. It will find weird strings that looks like keys or secrets. So I think that shouldn't be there and it's incredible what you can find. So that's an interesting part. Scan the JavaScript fives.
Okay, okay. So how can you avoid that, for example? Sounds like instead of leaving keys, maybe you have some build process. During the build process of your scripts, you're supposed to take off those keys or how do you do that? Exactly. So you can use tools like Gitleaks is a very famous one. It's on GitHub. Another link will live below. So Gitleaks, if you're an DevOps engineer, it's great to integrate into the CI process. That's what I do.
And that's basically taking that, configuring how do you want to scan, how back in the commits, how back in the tree of branches or whatever it is, and it just scans your Git repository or the entirety of Git. And one thing you find a lot of times developers, let's say you committed the secret for whatever reason and the tool finds it. What most of the developers do, they just commit another, they just push another commit that deletes the secret, right?
And that doesn't do it because it's still in the history. Git tracks your entire history if you don't remove it from there. And then you need to go through a process. There's yet another link, a BGP cleaner. You can find it on GitHub and that goes through the history and finds every commit that you put the secret in and it will just clean it from the history and push a new version of everything.
Or a more hacky way to do that is to just delete all commits and commit a new one from scratch without the key inside and then you don't have the history. But that's one way. And that tool, by the way, Gitleaks, if you just go, you can scan an organization. You can just take a random organization on GitHub, which is another huge place for secrets and flaws and things that people just push to the web because, well, open source code. You can scan an entire organization.
You can say, I want this organization. It will take all the repos, pull them locally or not, and try to scan them one by one through the entire tree of the Git history and find amazing stuff. So I run Gitleaks as part of my CI build if we go back. Oh, okay, good to know that. All right, so that's scanning Git. Okay, let's talk about maybe application design a bit. Okay, so let's say I got a node application. And this node application. Why node? I like node.
Okay. So in that node application, you got to admit, you like doing this Yarn install and then it takes a lot of time to install all those modules and then you feel that you did something meaningful, you know? So you like it, you know, Yarn install and then you're like, I'm doing something. You know, it's working. I hear my Macbook is shouting and it's crying during the process, you know? So I like it. So this is why I'm using node, just to make my Macbook cry. Okay. This makes sense to me.
That's the only reason. That's the first time I hear a good reason to write JavaScript. Yeah, other than that, it's famous. But, okay. So node, right? Using another application. Let's say I have my backend server. And let's say I'm using even ExpressJS, trying to, you know, talk to the broad audience, like to speak about the technologies that probably most people know. Okay. So let's say I got an ExpressJS. And for example, I need to call some APIs with a secret key. So let's take app key.
Okay. It doesn't really matter. App secret, app key. And I also have multiple environments. Okay. So I got development, staging and production. Right. So I got a backend application with written in node. And I got a secret, let's say environment and whatever. Now, I know as a developer, I can use maybe process and or whatever to reach out to the secret. Maybe if I inject the secret as an environment variable, I don't know.
So maybe you can, or maybe I'm going to say something you're going to be crazy about. Maybe I will use the secret. If it's a Docker file, I will, during the build of the image, I will inject the secret as an build argument. You're supposed to shout at me or something like that. So can you tell me which best practice you do? So if you got a Docker file, node application, you need to inject the secret that will be used in runtime. How do you do that? Like which method do you use and why?
Okay. Wait. So that was a long question with a lot to unpack. So let's roll all the way back and before we touch the node application and the fact that it's in a container, let's, you know what? Let's keep with the node. Let's break it. Yeah. Let's start with the node part of it, right? You're using Yarn, NPM, whatever it is to install a lot of open source or third party libraries that you brought in.
More often than not, that would be an open source project by someone out there on GitHub or somewhere else, right? You don't know what's going on there. By, you don't know what's going on. I mean, you're not sure what type of vulnerabilities you're bringing in. You probably don't know the developers. You're not sure what's going on. It's not a service you pay for. I mean, the entire, the majority of time you're not paying for it. You're bringing something open source.
You're not sure who it is, how popular it is, who's scanning it, what are the processes there. You have no idea. You're just pulling it in. It's good to send, scan for vulnerabilities. You can do it alone with another open source tool that you can find on GitHub. You can do it with lots of cyber security companies that that's what they do. They will help you scan it. In any case, you want to run that.
You want to scan your third party libraries before you deploy them to production because that may very well be the attack factor that can expose you to a hacker or to some kind of vulnerability. So that's the first. Another thing that you probably want to do, and now we're getting into the containers themselves or any, you know what, another step back. I think security misconfiguration and design configurations are two of the top 10 of OSP.
And without diving too deep into both of them, you want to design the application with security in mind, which means securing your APIs, making sure you're not falling into traps of access control. By that, I mean that you have a very strict limitation and segregation between what a certain type of user can do, what an admin can do, what, I don't know, a manager can do. There are things like segregation between different organizations.
You're probably serving data or holding data of different users. And those different users belong to different organizations and you never want to mix up their data. So you want to make sure that one cannot access a different type of data. And a lot of times different vulnerabilities like these can just appear because of bad design. So without getting into solutions, I'm just saying develop with that in mind. Now to the part of the containers you mentioned.
I wanted to bring up something that's called 12-factor app.net, I think, right? Or something like that. So that's kind of a list. It's beyond security. It's more of a design thing, but it's a way to design modern applications and one aspect of it is having all your secrets and parameters a part of the environment. By that, I mean that when the application starts running or before it starts running, it doesn't know where it lives.
It doesn't know if that's staging or production or whether a certain database endpoint is this or that. It just starts running and in that sense it's completely degenerate. And all of these types that change the environment and change the secrets and change how it's being handled are coming from the environment and that's injected to the container. There are many ways to do that. You can do that by environment variables, literally environment variables of the container in the Docker file.
You can do that as environment variables injected into the process. You can burn them as a file. I mean, not my typical use case, but you can burn them as a file that's either mounted to the container like using vaults sometimes. There's no way to do that. You just kind of take a file and mount it into. Kubernetes is used to use config maps and mount them into containers. Lots of ways to do that.
The point is use the environment to handle the application both in the secrets you use and the parameters. So it was a long one. I don't think I remember the end of it. Let's dive in back to the, let's say, Docker file. Why is it bad if I have a secret and I want to use the arg, you know, ARG command in Docker file and I want to put maybe the arg and then put the, I don't know, secret key as a build argument. Why is it bad to do that?
That's a perfect point. Okay. That reminds me, you're building a Docker file and you're putting it somewhere. It's going to be hosted on Docker registry, whether that's ECR, GCR, Docker Hub, whatever. Those things can be exposed as well. And that's another reason you probably don't want to burn those things whether as build arguments, I mean, build argument probably suggests you've burned something in the process of build.
And then when the container is wrapped and then pushed to the registry, it already holds your secrets. If you don't do that and someone manages to access your Docker Hub, maybe you pushed publicly without meaning to do so. Maybe someone got access in some way. I don't know what the layers found themselves to the internet. If you don't hold secrets on top of the container, then yes, parts of your applications are now exposed.
There are ways to handle that as well, like building binaries and so on. But your secrets are still protected because they will only be mounted to the container in runtime, right? So that's an answer to that, I think. So I think we got a rule of thumb for that. I think we got a rule of thumb, rule of thumb for that. If you feel that you need to maybe hold the secret in your Git or in your Docker image, you're probably doing something wrong.
So what you should do is inject your secret during runtime. So secrets and runtime, that should be the rule of thumb. You know, you say secret, you think runtime, right? You reminded me of yet another thing you can use. For example, on AWS, on ECR, ECR has this scanner feature that can scan the images. I think it was a women's name. Claire? Claire? I think Claire. So Claire, I think Claire, we'll check it out later. Open source scanner for Docker container, any type of container image.
And I think that's what ECR does under the hood. But in any case, you can use wherever you host. They probably have a scanner of their own that can scan the images. And they find secrets and keys and vulnerabilities. And that's yet another layer of security you can apply. Do you know why it's not working for Windows images? You know why? Because nobody wants to work with Windows images? No, it's because they're too big.
Every time when the image is too big, they're telling you, listen, it's too big. I'm not going to scan it. So if you've got, let's say, 10 gigabytes image and that can happen with Windows. I can't say easily, but that can happen. And in Windows, you have a 10 gigabytes Docker image, so those tools won't work. So let's inject at that joke here and say that if you're using arguments with secrets, you're probably doing something wrong. And that's exactly the same with Windows containers.
You're probably messing with containers. You know what? You should... Yeah, you know, sometimes there are those who still use.NET. And by the way,.NET 7 is amazing. I don't know if you know that. I was joking, of course. I'm just saying if your cloud provider or service provider doesn't help you scan the images, you can use Clare or any other open source or no, not open source one. In any case, you can scan them.
The fact that they're big is probably a limitation of the service, not a limitation for you as an organization. You want to scan them just as well. Okay, so again, I say secrets are for runtime. All right, so let's talk again about this runtime. Before we run out of time, let's talk about the runtime, all right? Okay. Okay, so let's say I got my application, it's up and running. I want to ask you something about... Let's say I want to update a secret.
Okay. Now, some might say, and I think that's a very cool question. I don't know how I thought of it, you know? But some might say, listen, secrets shouldn't be changed on runtime, especially on production. So if you want to change a secret, and let's say my secrets are fetched from AWS, Parameter Store, or Secrets Manager, or Vault, or... I don't care, I don't... Okay. And some might say, each time you want to change a secret or a parameter value, you must redeploy the application.
Some might say, you can change a parameter, and then the application will reload the secret automatically. Right. Which paradigm, which... How do you feel about that? Or maybe it depends, again. I think it's one of those things that... First of all, it's an approach. So it's usually more of an application leader's approach. I don't think it's up to me to decide. My approach... What do you like? You're here. I like things that rotate, but regardless, you just said that... No, not even rotate.
Let's say not rotate. Like, I want to change a secret instantly. I found a bug in production, and the API key that I initially injected in the CI CD was wrong. I don't want to initiate another CI CD process. I want to change it instantly. I want to change it now. I prefer not doing that. I prefer not doing that. I feel like the ability... I feel like it's a nice thing to have, the ability to change it instantly.
But in my experience, when I did those things, I was hurting so many col... I was creating collateral damage in a way that convinced me to keep the rolling updates in place. And if my CI is stopping me because it's too long, then the focus needs to be on shortening the CI pipeline rather than making... or creating the ability, rather, for developers to change things while they're live. Especially for me, who's working with tons of Lambda functions.
If I change one thing, I might be seeing collateral damage in many other functions. I rather not do that. I rather have the CI run again, scan everything, make sure the end-to-end tests run, etc., etc. Because this is kind of skipping the entire process and the entire idea of QA or automated testing, or whatever it is. I don't like that. Okay, so I like that. We've got a concrete decision about it. You're saying we've got to re-trigger the whole CI-CD process.
If it's too long, let's focus on that. But don't skip the CI-CD process. That's my approach at least, yes. Okay, I wonder, because I'm sure there are many developers and VPR and diesel are out there saying, listen, sometimes it's a production, it's a production issue, it's a bug fix. It's got to be fixed within a few seconds. No problem. No problem. Then make your CI ready to deploy a hot fix. A hot fix can mean you skip certain areas.
Maybe you have an end-to-end test that runs for 30 minutes and you want to skip that. That's, well, not fine. I like that. Okay, so this solution that you're saying right now, this sounds like a hybrid solution. It's not saying, skip everything, do it behind the scene, and it's not saying, let's re-initiate the whole CI-CD process, but be prepared for, if you want to do a bug fix, sure, just prepare a CI-CD process for all that. Try to be structured.
I mean, if you are now changing a secret in production, maybe that's in Vault or a secret manager of any type of kind, you might be hurting production without understanding what you're doing. You're not a machine. You didn't run the CI process. You didn't run any type of tests in your head. I mean, you are a machine. No, but seriously, I totally like it. I totally agree with it, and I like the idea. I think it should be implemented across many pipelines. Prepare for the bug fix.
Prepare for doing the worst. No, wish for the best. Prepare for the worst. So this is what we're here for, right? That's a good ending mark. Okay. By the way, before we end, because we got to end in a minute or two, do you hear the cat in the background? No. No? Okay, so we got a cat here in the background. Sounds amazing. Anyways, to our final corner, okay? Yeah. So as you like to ask, you always ask this question, which technologies did you encounter this week, if any, or challenge?
Challenge your technology. You pick one. Or just a cool project. All just a cool project. Okay. Project technology or challenge or anything else. All right, cool. I think this, we've recorded like 20 something minutes, and I think it's going to take two hours to just find the links we mentioned in 20 minutes. It will take one minute. I'll add yet another link. Yesterday, I've seen a project by AWS that is basically a set of Python scripts.
I think that they're called Harden EKS, which means you can run a set of rules. You can obviously filter them, what you like and what you don't like, but it runs against your EKS and kind of outputs a report that you can use to Harden EKS, not use wild card roles, secure your ingress permissions, et cetera, et cetera. There are like dozens of rules there. So pretty cool project, and I'll put the link below. And I think any user of EKS should check it out. Okay. Sounds cool.
I'll pass on mine because I think we are running out of time, and I don't want to make it any longer. So I just said I had a cool week, and you'll have to endure that. All right. That's even better. Yeah. Okay. So thanks everyone, especially for those who are not here. And thank you. Next week. Namaste. Thank you.