Hey, everybody, welcome to another thrilling episode of Adventures and Develops. I'm your host today, Will Button and joining me today is adar Shaw. Adarsh welcome, Thank you, thanks for having me my pleasure. So do you want to give us a quick introduction and tell us tell us what we're going to talk about today. Sure. My name is Adashah. As you mentioned, I'm the founder and CEO at compus Est. I've basically played a lot of different roles over the years, from a developer to an architect, to
a consultant to technical product owner, mentor advisor. But I'm an engineer at HAUT who loves to build stuff that helps business technology, stuff that helps run business. And right now I have built up product as part of a company that I founded, and also do some consulting and advising around the DevOps and infrastructure as code space right on. So one of the things, and I
want to jump in because I'm really excited about this. One of the things that you put in one of the articles I read from you was talking about not infrastructure's code, but environment is code. Do you want to bring everybody up to speed on that? Yeah, definitely. So what I and some of the colleagues that I've worked with realized over the years is like, an infrastructure code is great, right, because it helps us automate provisioning of various
infrastructure resources on cloud providers on premises. But what we realize is what people need is an entire environment. Right. Just having your networking or just having your EC two instances or what have you, is not gonna give you everything
you need to run your application. So what I and my teams have done in the past is like kind of go through this painful process of like, right, pipelines that run stuff, connect various pieces of these if you think about it as lego pieces, like bring them together and kind of those things get really complicated and very difficult to manage. So what I kind of thinking about this more and more in the last few years, I was like, you know what I mean, there has to be a better way of bringing
these pieces together. So started thinking of this as a concept and eventually named
it as environment as code. It's just an abstraction over infrastructure is code that basically provides a way of defining an entire infrastructure environment, right, not just a single component or a resource, but also it manages state for the entire environment, including the dependencies between where resources, and also supports best practices, right, Like when you look at infrastructure is code, one way of getting
this whole environment is by writing this monolith infrastructure is code where you can put everything together. Right, that's not a good practice, right, like thinking about loose coupling and things like that. So, yeah, that's kind of what I call it. Environment is code. But basically it allows teams to
deliver entire environment instead of individual infrastructure resources rapidly and reliabily at scale. Yeah, whenever I just first read the term, it was like this huge light bulb went off, because I think the term itself is so much more descriptive of what we're trying to accomplish many times with infrastructure's code, because the infrastructure itself is just a component of the overall environment, right, and each piece,
each individual piece of the infrastructure is pretty meaningless unless it's couple with all the different pieces of that environment. And like you mentioned, I've built many environments using that monolith approach, and then you have to update one dependency in there or change out one of the components. And it follows very much along
the lines of building a monolith software application. You know, we're just making small changes, becomes this task that you dread and procrastinate and hope that you don't have to do it, knowing full well that you're going to have to anyway. Yeah, exactly, and some of the other things. Right.
So some of the principles that we talk about it like immutability, right, So you want to be able to easily provision a brand new set of Like let's say, if we talk about servers, rather than updating and keeping a long lasting server in case of this, applying that same principle to an entire environment, right, So why keep your various components living for long? Can
you actually replicate an entire environment rather than just an individual component. So for someone listening who's liking this idea and wants to explore it further, what do you think the first steps to migrating from infrastructure's code to environments as code is like, yeah, so, I mean, like I said, I think it's just an abstraction. So thinking about these various pieces and how they get together. So what we did is we essentially to keep it really simple,
is created a simple Yamel format. Again, this is new, so I'm like a lot of that I'm saying, at least from my perspective what I've seen. I haven't seen anyone talk about it this way, but it's new. So what we did is we provide a very simple format, a yamal format that actually specify these individual resources or components and how they connect with dependencies. I have an article that I think I've shared that talks about the principles
and how what this means. Yeah, I mean it's it's like I said early stage, So I would say, like something that allows you to create an abstraction on top of infrastructure's code, that gives you an entire environment rather than individual component, right, because a lot of people, like I said, write pipelines right to do this, and then you have to think about things like, hey, how do I provision stuff? But how do I
tear down right? Because teardown would be totally totally opposite graph right, So you have to tear down those leaf nodes first, and you just write all this complex logic if you've changed. So defining an entire environment and easy format that provides ways to connect those lego pieces if you talk about it, and then that takes and enterprets like into visions, those individual pieces so that you
get an entire environment. One of the things you put in that article you mentioned in there about keeping everything in source code, but then you added to that a little thing that I know I've slipped up a lot in the past doing this, and I've encountered situations where other people have done the same, like those one off scripts that you only run ever once in a while,
about how important it is to get those into source control as well. Yeah, definitely, I mean I think, yeah, talking about I think you're talking about the patterns and practices in general for infrastructures, code environment because it doesn't matter. So yeah, like I think, yeah, same, yeah, right in the situation where like hey, we just ran the script and you know, forget about it, right, like you it's on someone's machine and then you try to find who did that and how it was done.
So yeah, I mean source control being a place where you can track things, have auditing, right who pushed so you can talk to the person about it. And so yeah, like I think people say, like, hey, just put your terraform code or blow me code or configuration know, everything including Bash scripts that are lying stuff like you know that you use once in a while, everything needs to go there. Yeah, what about documentation? Do you like documentation in the code itself or do you have a preferred tool
like Confluence or something else for keeping track of docs? Yeah? Yeah, I mean so one of the things is docs is like a lot of people say this, like, hey, you do infrastructures code or like everything is in code, why do you white you need documentation right? And it's just I mean, yes, you don't. You need less documentation, but you
still need documentation right, And in general documentation it's hard I understand. But so like like a lot of people are like and I used to at one point, I was like, hey, we don't need documentation because you know everything is codified, but you still need right and talking about documentation right, quality documentation, making it a well bill where let's say there is an error, provide a link that can take a person who's actually looking at it directly
to that. And also it's just providing a way to keep it updated, right, so like I can draw a diagram or like write documentation and just you know, like hey, it's out of date, it's not going to be useful because it's actually giving you wrong information. So yeah, like I prefer obviously have naming things better in your code, right, because that tells you what it is. But definitely have some documentation that's external to the code that will get you started. Even it could be like, hey, here's
an index of things that you look at. This is how you onboard a team member or a customer or a user of your system. Also look at things like run books, right, that helps you look at specific scenarios around troubleshooting and things like that. Another important aspect about documentation is like, not everything is automated, and you actually don't need to automate everything right away, right, But you hit that second or third time you're trying to do the
same thing. Maybe document at first if it things are not automated, and then use that run book or what have you to actually automate things later, right, because that way there is a progression into getting to that automated state. Yeah. I think that's a really good point, Like everything doesn't have to be automated because some things, like in my experience, one of the
common scenarios I've hit over and over again. I work a lot with early stage startups, and so we'll build stuff and then that will fail and it just gets scrapped and never comes back to life. And so when I think about automating things, you know, I try to automate the things that are going to survive into the future, and it seems to save a lot of
time. And I know that's there's a tendency to automate everything, but for some things that just are going to fail fast and then disappear and never be brought back to life again, you can actually move a lot faster if you don't try to automate those. Yeah, yeah, I mean exactly. Especially let's say you have there is an error in one of your environments, and then it's like fixing or troubleshooting, right, even documenting those steps and then
eventually using that documentation to actually have a fix. Right. Because so whenever you talk about environments as code, is there a right size of team or environment where it starts to make sense to implement this, Yeah, And I think, yeah, it's like if you have a very simple setup, Yeah, it doesn't make sense to start complicating things because obviously with this like that
also comes a little bit of complication. So yeah, I mean I think I don't have a number for that, but I would say it's like a typical setup where you have various layers of your environments, right, so your networking layer, your platform layer, your application infrastructure layer, right like databases
and things like that. Once you get into those kind of things, and an application is not like your setup is not as straightforward as let's say I am just provisioning an easy to instance or like just you know, using lambdas
or what have you, right, which is like very straightforward. Yeah, I would start thinking about it because the other way to look at it as well is that, hey, let's start with writing these pipelines, right that provision these to start with, and then once you start feeling that pain of like adding the third component on the fourth and I was like, okay,
now this is getting like how do we tear it down? Then start looking with anything in software, right, don't overcomplicate to start with, But once you start feeling those pain think about a way to abtract it, which makes sense for provisioning the entire environment. I think we can come up with a new buzzword there. Instead of test driven development, we can call it pain driven development. I love it. Yeah, that's awesome. It seems to
be my big motivator. Yeah, so you've got you've got some thoughts here on testing. What is what are your what's your high level overview of the type of testing that you need to have wrapped around us? Yeah, so testing for infrastructure's code environment code as in I mean thinking about the test pyrament that we keep talking about when we talk about application testing, right, so kind of similar idea, but obviously it's different, right, Like, writing
unit tests for your infrastructure's code is going to be complex. I mean, there are tools that allow you to whether you need it is another question, right, So let's kind of Yeah, so let's kind of talk about a
typical test pyramit for something like this. I start with at the bottom, it's like do static analysis, right, So if you're doing terraform, like do things like terraform validate, you have lend or maybe you're using Puppet or what have you, Like, all of these tools have some static analysis tool.
When you get into unit testing, it really depends. What I tend to suggest is like if it's something like terraform that is declarative that actually allows you to define the desired state, and it's Terraform's responsibility to actually get you to that desired state versus more like your conditionals and things like that, where you need to write some complex unit tests and things like that. I would say you don't need unit tests when you're talking about declarative stuff because you are
not writing that logic to actually make something happen. You're just defining a desired state there. And then it's the tools shop, right, so if there's a failure within the tool itself, it'll tell you that, hey, you can't get you to that desa state. So like most of the cases, you don't have those those complex logic right. But then if you move up this pyramid, you talk about integration tests, right, which makes sense in
this case. Could be like in your provisioning pipeline or whatever you're using, you can bring up an epheneral environment, a temporary environment and then run some
tests against that. So let's say you need a setup with networking and like some platforms like EKS and what have you, you can run some basic integration tests to see like if those resources have the things like for example, running to make sure like hey, none of your S three buckets are public, for example, So you do those things ahead of time rather than when you actually provision that production environment, right, so you don't want to be finding
those issues late. Right. And then the last one there is like which I have useful, is like the best pace to test your infrastructure or environment that you have provisioned is by running deploying a dummy app. So let's say you're provisioning a Kuberne dis cluster. You can run like one simple dummy app that is close to what you would be typically running and then run some smoke tests just to see, like, you know, if you can get there.
So again, it's a test bit of it. So at the bottom where you're static analysis and things like that, you'll have more of those tests rather than and when you as you go up the pyramid, you have less of these tests because they are more expensive and time consuming. Yeah. Yeah, exactly. And I think one of the things you said in there that
I really want to call attention to is talking about unit tests. And with a declarative type framework like Terraform, there's really not a lot of value in unit tests because at that point you're testing like did the tool actually do its job? And a tool like Terraform already has tests to cover that, so you can you can save yourself a lot of time and effort by just acknowledging what tests the tool has the test that the tool is doing its job.
Right. But yeah, I mean if you're writing let's say Bash maybe like when you have all this logic, then in that case that's different. Yeah. So how do you deal with baking security into the environment? Yeah, and that's the thing a lot of people, I mean, that's a very important aspect, right, and a lot of people tend to like, hey, what do we need to care about security here? But yeah, I
mean I think a few different things to consider. One is you with with these tools, right, since it's automated, you need to give these languages, tools or whatever it's running access to your aid abous azure or whatever you're using to actually provision these right. So the easiest thing is give root access everything. You can do everything, and then you know if it's compromised.
Right. So I think one of the things is using a role based access control or trying to limit the surface area of attack, right and looking at if there are various layers of your infrastructure environment, maybe having different roles for those that give limited access to doing those things, because specially when you have production running, one credential gets compromised or so, like you can compromise basically give and I've had those issues that I've gone through it with multiple customers,
so that'll just stop everything, right, like the whole thing is compromised, rather than saying like, hey there's limited access, so even if things are compromised, the surface area is That's another thing you can do is if you use tools like hashikore Vault or I think even Secrets Manager AWS they have a rotation policy, right, so you can you can have these dynamic credentials created that then used. So again it's time based they expire, right, so
like you're not exposing it for a long time. But in general, you need secrets right when you are provisioning your environments or infrastructure. So keep it in in general, like just basics, right, keep it in some kind of a good secrets management tool which which stores that and then also looking at things like security scanning, so basically running those as part of your provisioning process
early on where you can spin up a firmeral environments. There are tools like CES Benchmark, the Amazon airfs as inspector and things like that that allows you to run these security tests or scanning to see like if you have things exposed right like is your S three buket public somehow, or like your databases are exposed and things like that, so it'll also check for other vulnerabilities. So yeah, I mean these are some of the things. Writing some common security
tests like using inspect and things like that also help. Like where it could be like just any infrastructure environment you provision, we just run these kind of common security tests to see like your network is secured, your resources that secure
would also be a good idea. Yeah, And I think that's one of the one of the possibly hidden benefits to using this approach that you've got of the environment the environment based approach, because like if you have this monolith infrastructure and you have security groups in there, you've got to think about this security group, say okay, this security group they need access to my sequel and elastic search and the reddest cache, and then there's an API serve, and
there's all these different rules and exceptions and all that kind of stuff in there that creates this huge vulnerability footprint. But if you break it down like you're recommending here, where everything is just a component, then you can when you're building out that component, you can say, well, this component needs to talk to postgrass, so we'll open the postgrass port and you don't have to
open anything else up. And then when you tear it down, you know that you can safely tear down that security group because you know the one and only place that it was ever used. Yeah, exactly. And one of the things that I talk about when I talk about environment, it's called is
like blueprints or templates for your entire environment. Right, So if you build these blueprints using best practices, keeping security compliance and other best practices in mind, essentially you can you can templatize it and then kind of use that and say like, hey, we did this and this is like certified or you know, by our team, and then you can share it with other teams
within your arc. So that gives you that versus if you look at like, you know, doing it using pipeline or other things, it's very different, difficult to like, you know, creates these templates for the entire environment. Yeah, because once you change the pipeline, everything downstream is now in an unknown state. Yeah. Plus these pipelines are not meant to run these things. They are more generic and then like you write all the logic to
make whatever happen, right, so it just gets complicated. So, speaking of the pipeline, whenever you do the infrastructure and build this out, what are the pros and cons of using an autumn using a tool like jink injer circle CI to deploy this out versus writing it and building it and deploying it
manually. Yeah, yeah, I mean it's it's a very important aspect, especially when again, one thing I say, like, if you have a very simple setup you're getting started, maybe you don't need it right away because let's say you're the only developer, like you know, you have a very
small theme. But as you as you grow and you look at provisioning and automating these things, having this shared auto execution environment, right, I call it where things are set up which are replicated, right, so hey it works on my machine problem right, Like like hey I ran this, but oh you had this version of terraform or this version of that, right,
so you can avoid that. And basically I feel like using any of these pipelines is one way, right or that have this common setup that you use every time, right, So it's not your machine, it's something that has all the tools and right versions and things like that. That definitely gives that. It also gives visibility, especially if you're running these tools a lot of times. You know they manage state, and if you've run a lot of
times things locally, you have local state and stuff like that. Moving all that to a ship place helps because then you know, if I'm doing something one of my team members is running, they all go to the same place to do that. Also, there's another way, gitups, which is more and more talked about recently, but essentially how icy GitOps is. Basically, you take your infrastructure's code, add a workflow to it, which is typically your PR process which all the developers are used to. Is like, hey,
create a PR, someone approves it. Once it's approved and merged to your main branch, it automatically provisions that so now you don't need to push a button or do anything right, but it also has this flow which I feel is a nice flow off, like you know PR process, right, so you can have people approve it, take a look commented before you merge
it. And then I also feel in there that getups is like this control loop logic right that talks about this detect th drift and kind of bringing back the desired state to actual state to desired state if things go out of sync. So I feel like that getop's way is another way of looking at it. When you're running your infrastructure's code or environment code, but using like the Shared Environment