Kubernetes Schema Validation Tools with Eyar Zilberman - DevOps 227

Welcome everyone to another episode of Adventures in dev Ops. I'm your host for today, Will Button, and we have our new panelists with us. We have Jonathan Halm Hello, hello, and Jillian Rowe Hello everybody. And then we've got our guest today, we have er Zoberman.

How are you doing?

R Hi. Nice to meet you all and excited to be here.

Well, we're excited to have you. You want to give us a little introduction about yourself?

Yeah, sure, So my name is Yal and I am leading the product at the company named the three.

In at the three, well, I think a company is prevent configurations.

And fun fact, you actually also my co founder at episode number seventy six, so I give a reference to this episode and I won't go into details about exactly how we're doing that because it's plain it's there. Besides leading the product Editree, I'm also leading the local community of get up in Tel Aviv, which is the biggest one in the world over twenty five under the users and beside it and just love development.

I actually was a developer before I.

Went into the tree and as a product leader. And another fun fact, I actually have a lot of degree so I have nothing to do with development. It's all learned and I actually love early love code, and this is how I got into this space.

So you said you have a lot of degree.

Yeah, this is correct. I actually have a lot of degree. And I was supposed to be a.

Lawyer, and so the prospect of being a lawyer was so horrible you decided, no, I'm going to work in tech instead went.

Something like that.

Basically, like, while I dealt with law, I always loved the technology, so I did like law and technology stuff. Basically, it was a lot of open source licensing because a lot of people never never really understood what is the open source and open source people never understood what is low So I was in the middle.

There was able to talk with both sides. But during this.

Process I actually fell in love with the technology and then decided it is part of the open source it's much more interesting. So I got into the open source to developing by myself and then went into the process of being a developer. So I have all degree, it's somewhere on the wall, but I'm not using it.

So it's not that you thought law was too simple and you wanted a better challenge. You wanted something more complicated like Kuberneties to work with. That wasn't the thought process?

Yeah, something like that, get out of your comfort comfort zone?

Are you like certified in law? And you sent out kind of season disiness platters because I think that would really.

Come in me And sometimes I prefer not to do that because again I did it like a few years ago, so I'm not up to date to all the new rules and stuff like that.

But you're up to date with Kubernetes, right, this, this is correct? Yeah, what's the latest new feature here? Excited about?

Wow?

We promise not exactly too many for me to mention.

Cool, But you did write an article that we've got here on why you need to use kubernating scheme of validation tools, and you actually looked at two different ways of doing that, Cuba vallel and Cube conform. What was the motivation behind it? I'm assuming that there's like a backstory here of where something happened and you were like, oh my god, we cannot go through this again.

Yeah.

So actually there's also a third option. It's like actually doing it with cube cattle, and so the backstory is that ed the tree. Like I said, we're helping companies prevent coubernet dismiss configurations. We're doing that by scanning the manifest files and giving them indication if it's up to the standards that was defined by the organization Police also called.

And something big that we.

Got is that a lot of people thought told us that it's passing the policy, but it's still not a valid Kubernetes file. How come because I know someone forgot to configure it correctly and instead of calling it, I know API version with version in a capital letter, it's all smaller or something like that. So it's still passing

Was talent with our corner of the internet there, didn't you?

Yeah, like someone can hide a body and the results about how to do schemu vilation with cube cattle.

You can hide a body there and the resultso no one will find it. Nobody will find it, No one's looking exactly.

It's going to be your new title. I was just wondering as you were describing these, are any of these integrated with HELM or these if you're writing your Kubernator's configuration files manually or through some.

Other that's a really good that's a really good question.

So if you think about it, and basically it's also a Combernetus manifest and in dead we're also rendering Kubernetus manifests, So it doesn't matter like all of them will walk with the head. It's only a matter of do we have like a native integration that it will be connected to M directly or another way to do that is to render the manifest with HELM and then passing it to one of those tools.

Yeah, that's an interesting way of doing it. Just just have Holme render it for.

You and then throw it off to one of exc exactly like people forget people forget that you exactly that helm is actually in the end, there's a Kubernetus benefit that's generating, and this is what's getting pushed to your cluster.

Usually you don't see it because it's pushing it directly, but if you do ham template, you.

Will see the file itself that is pushing. Cool.

Now I have an extra step in my make files to.

Add I think that's a great that's a great point. Though, where do you recommend people do the validation checking out?

Yeah, so just for the people that listening and didn't read the article, we.

Just say that good use.

If you have a scheme of validation errow, it will get cut in the end because basically, when you try to deploy it your Kubernets cluster, Kubernets with throw an errow to tell you that it's an invalid a combneti is file.

That's all good.

The problem is that you want to catch those arrows as soon as possible. You want to shift them left. You don't want to wait until you try to deploy it. You want to catch them when someone is submitting them. And that's the problem. Because with cube cattle there's something that is called like it's a dry run flag that you can say something something applied minus dry run and then it will connect to your cluster. It will check if it's a valid file. If it's a valid file,

It's only I.

Would say it's like cubivalve with superpowers with the cube and phone. And the other way for you to do it is actually with a tree. So with a tree we, like I said, it was an issue that we add, so we also added those capabilities to our tool. And if you are checking for policies, there's also pre acquisites

So this is also something that you can do.

I will also say that another thing that is interesting and I wrote in the article, is that you have another flag with cube cattle.

So we have two modes. You have several mode and you have client modes.

Basically you can check both of them requiring you to have a connection to a cluster. Something is think that I discovered was that actually there's an open bug in the Kubernettis project, and the open flag is saying yeah, yeah, yeah, among those one thousand bucks that are opened there, and this open bug is actually saying that this is not the expected results.

If you're using.

The flag dry run but on the buts a client mode, it should not need to have a connection to a cluster. But right now it's not walking, so it's still requiring you to have a connection to a cluster. Another interesting thing, and this is also I explained in the article, is that there is a discrepancy between the validation that I've done on the client side and the validations that are done on the SEVI side if you're using cube cutted.

So answer question, well, now, go ahead, go ahead, and then I'll argue with you.

That's fine.

So just to wrap it up, the best way to do that is as soon as possible, you should run those validations across the entire process from your local environment through ci CD and just before you're going to deploy it or any other automation process that you have staging, production whatever, do is as soon as possible, and do it all the time.

I actually wanted to argue with you a little bit on a point about not having access to a cluster while you're doing these validations. I would think you would need access to a cluster, because what if I'm doing like no affinity is or okay, that's the only case that I can think of, actually is when I have not affiinity. So I don't have a real strong case

So think about it like in big organizations that you have a lot of developers and so usually we're saying CIS, but we need to remember the CICD are two different steps, and there are a lot of organizations that I'm familiar with that the CI step is taking X amount of time and only then coming to the city step. So during the CI step that people keep changing the manifest,

Only when on the city.

Step you want to have a connection, you have to have a connection to your cluster. So if you separate those steps, which usually happening in big organizations, the CI step don't have connection to your cluster.

So I'm looking through your article and some of the you have this nice little table that compares cubevel and cup perform against client mode and server mode of cup pedal and what things were caught and what it didn't. And I'm clicking on some of these here, and it looks to me like in some of these cases it's looking more for syntactic validity than contextual validity. I don't know if that's the right phraseology there, But for example, I look at the label value and it's the wrong.

So that's a good question.

So basically, there are different steps of validations that you need to pass if you want to have a valid file. So first of all, let's think about it like on the general air view. You want to make sure that all your cuberneties files have to be a valid diamined file.

That's first of all.

After that they have to be a valid Kubernetes file, which means they need to follow a specific structure. After that, the values inside those files need to be valid, and different steps or different tools will catch different errows that I just mentioned. So with a tree, you will catch all theres, will make sure that it's a valid diamond file, will make sure that it's a valid tubunetifier. Will also make sure that the value is avalid and we cube valid.

Left to the right, right to the left, to the.

Left to left in Hebrew, also or do you shift right since you're the other.

We read the opposite. That's the problem, you know, that's why the confusion.

We're really from We're really from right to left, So like make.

No sense the Japanese shift up, cultural adventures and DevOps. I had a great question, and now I completely lost it.

You want to shop to come back?

I think, okay, Well, I was just thinking, you know, like this whole idea of okay, we can say that it's a valid YAMO file and the valid Kubernetes file, but doesn't make sense. And to me that's always been like such an interesting problem, like one of the more interesting problems, especially because my background is high performance computing.

Sense or not.

That might be the only way to do it is have like a massive decision tree that nobody actually understands that says yes or no.

I think you just described Kubernetes exactly.

Yeah, a little bit.

Okay, I remember my question. I'm curious what does your workflow look like when you're working on Kubernetes manifests? Do you run these tools in your editor for example on save? Do you use githooks? Do you use CI pipelines? What does your setup look like? How do you do this in practice?

Well, I'm biased, a musing I own, but yeah, but I'm telling you so what I usually see that people are doing this is why we credit this tool, is that they understand the value and they're trying to shift.

It left right.

You're trying to shift it left and they're doing it with pre committos. That's one. Then it's implementing inside the CI. The problem is that you need to implement a lot of tooling in order to get those simple vialidations that I just described. So you need to have volunteer for your YAM. Fine, you have a to have a cubival or cup performed for kubernettes. And then you need to have some way to actually do the policy checks, which can be performed with different tools that I have to

To do that.

And this is why we build a tree.

We're trying to do it in one tool, make it simple, make it fun so you can it's a sea light tool, so we actually enforced it.

Or you can put it everywhere you want.

You can put it on your local environment, you can put it in your CI, you can put it in your city, you can put it everywhere and it will do all those validation for you out of the box and a really simple and easy way.

That's very cool. And is it all open source?

Yes? Yes?

And again like yeah, so there is a magic sauce in the tree, Like it's not. We don't have a secret API. We are not doing something that like every developer can do. What we're doing it, and we are totally okay with that. And the cool part is that we're just trying to make it much more simple for you, so you don't need to do it by yourself. So you don't need to configure this plit commit and you don't need to configure this and integration by the way soever henp like, and so we can do it natively.

That all the stuff that we're doing will give you a value as a user.

That's very cool. You said something I didn't quite touch. There's a plug in for something, was it, Homer?

Was it?

Yeah?

Yeah, yeah, So you asked about the nice so I

Cool. Does it integrate with like any of the code editors too, Like will it tell me in nice big red leaders, because like I really need those reds telling me, but I'm doing something stupid.

If not, it's it's on the road map.

It's only on the road map because we really believe that we need to give this feedback about the validation as soon as possible, and on the road BAMP is to also put it inside your ID and if it's possible also in when you think about doing a misconfiguration to also be integrated there inside your head, you get like.

A buzz what do you call that? CRD?

Not sure? Not sure? We need to think about the name for that.

So I'm really curious about how this works with Helm because obviously Helm isn't purely deterministic in the sense that depending on what values values you provide, you could have an infant possibility of actual kubernes manifests to come out. How do you handle that? I mean, for example, I'm thinking of the chart testing or CT tool. I don't know if you're familiar with that, but it lets you define like a list of You could give it a directly full of values gamble files and it will just

So again, this is a really good question, but we need to remember in the end of.

Every helm file there is a couberneties file, so we are not checking the value file separately and the child files separately. What we're doing is that we're rendering it together and then we're running the checks on top of it. So in the end it's just a manifest file that is random from hand value and then child that is combined together. So it doesn't really matter how you do the templating.

From the all side.

You can use which key invirues that you want, because in the end to will be translated into a Kubernettis file. So we're just running it on the end result, which is the coupnettis file itself.

But if by values, suppose I have one value file that says ingress true and one it says ingress falls. That could help a completely different manifests completely different resources to find, and I might want to validate both versions. Does your helm plug in automate that for me? Or do I just need to have two lines in my sea ice script that says run it this way and also run it that way.

So if i'd the sun correctly, you're asking if I can run it in if I can have like two different policies because I have different permutations for the same held file.

Yeah, I mean, so I suppose I have a homestart that just deploys WordPress or whatever, and in one variation one of my configurations, say, disables the ingress, So I'm no longer creating the ingress a resource in my in my output, I'm not setting you know, several different things that might not be created. I'm not creating an st CL certificate and so on. My output manifest is going to be significantly smaller with fewer resources in it than if I had enabled ingress. And maybe I want to

Yeah, So basically, again it doesn't matter like we revitedate both versions. So there is a logic insight your code that will trigger one of them. Correct, So the version that is triggered, this is also what will be passed to the tree. And this is what we also be validated, and we'll give you the indication of its passingle failing the same that the same this mechanism that's triggating your.

Helm is the same one that will be passed to the tree.

Yeah. So the validate runs on the like helme install or helm up.

Grat command right exactly exactly.

Not beforehand. Then how are you going to integrate it with an editor?

With the editor?

Yeah, because if it's in an editor, it's before that helme install.

You're right, And this is a challenge. This is something that went into solved by don't have all the answers right now. This is something that we're working on.

That's interesting. That's where you need the decision tree.

Probably, this is why we call the tree.

Well, I've been seeing people have validating their values file also with an additional Jason Schema, and it seems like you will kind of work something like that out to sort of them have these trees that are like, oh, if you have a Boolean value, it should you know, it should track for both the true and the falls and these kind of things. But I don't know. I'm glad you're building.

It and not me.

That's very cool.

So you're right, I also saw it. You can do it with Jason GiMA is sorry, you can do it with Jason Schema. The problem is it's taking it's a lot of folk to do that, and also actually it's taken a lot of maintaining to make sure that it's always up to date, which is more hardened than just writing it. But it's not that common that people this is the best practice, but it's not that common that

That's true. I tend to just cross my fingers and pray on all the times that I commit to get help.

Yeah, so I think one of the things that was cool in your article here, because I know in my experience a lot of pushback I've experienced in trying to implement different solutions like this is how much time it takes or how much you know, people don't want to do it because they have this this idea that it's going to slow them down. But you actually did quite a bit of benchmarking on this right to see exactly what to slow down or impact would be.

Yeah, so this is something that it was interesting to me to see because why I check the different possibilities about how can I have how can actually have overcome the problem of scheme of validation. I noticed that when I'm doing it with cube cattle and I'm doing it with the salvable actually taking a lot of time to

It was so just to.

Give you the summer of that cube conform is doing it the best way. It's actually giving the results really really fast. After that you have cube revalve. That's also giving the result fast. Again, it's like on milli seconds for a regular usage, not when you're trying to scan one of the couplets files. So as a user you won't actually notice that, so you can say that it's almost the same when you're running it with CU cuttle

It's just going to take a little bit longer.

So if we think about it, we just said that cube cuttle server mode it's the best validation, so we don't really have an excuse why not to do that because it's not going to add too much time to your deployment process or something like that. The only issue with doing it is that it's requiring you to have a connection to a cluster, and as we already mentioned,

It be possible to run the server mode test against a test server like say running in kind or mini cube or something like that, or does it really need to be your production server with all your existing crds and everything is tolled.

Yeah, perfect question.

So you can do it with Minicube and then you can do it also in the CI or whatever.

But then you need to remember it's have to have the same environment like your production.

So if you have a name space that exists on production but don't exist on Mini Cube, it will fail because you try to deploy a file, you'll tell you, oh, I don't know this name space which is called Jonathan or whatever, because you have it on production. So it's a very tile but it will fail your your failure. It will fail on the CI. So this is something

Yeah, I think we could argue forever about like mocking

Yeah.

Yeah, it's a huge fool by itself just to sink everything, like to think this is something that is going to be lost somewhere and someone's going to forget about it, and then it's going to annoy a developer really, really, really really because you don't know why he's getting this validation errow because it's like, I don't know what to do with that.

And then there's.

Devils guy that forgot to actually sink the mini cube with that, you know it's going to fall in between the correct somewhere for sure.

I'm old. I'm going to start using this.

Tool me too.

Do you have a GitHub actions for it? Can I just tuck that up right now?

And so actually, so I have an example in the ouducts. We have an example about how to implement this side a guit ub action workflow. We still don't have a git up action per se. It's something that we will build soon. It's just the amount of integration that we need to build is just enormous.

So yeah, yeah.

So we need to have like a secret CIO and you need to have a ID integration, and you need to have a hand plug in. So it's something that we keep walking on. And by the way, we also have like an open issue on that in our Gita propository. So if someone want to suggest another integration, feel free because this is something that we always keep updating. For example, someone said like, hey, we need you need to have a homeproove. You need to be installed with homepoop, not

Yeah, it is cool. What's the business model this company is employing? Is there is there a commercial version of the software available or do there sell other commercial products?

How does just fit into that?

Because system Yeah, So, like I mentioned, I started as a developer and when we thought about this solution we want, we had one agenda, and it's to make sure the developers will enjoy using this tool and it'll be useful also without paying for you because like I said, you can always be the by you some So our goal is not to convert a single developer or a small team or team of tender velperlse. Our goal is to convert or to monetize big organizations that appreciate what they're

Also on this kind of show. No, no, I'm sorry about that.

I'm sorry, okay, So I'll give you Okay, So we said the number for one thousand because we know that people should not.

Pass it, not because you can't. You can't pass it right on.

A regular basic if you want to use the tool and get the value. There's no reason for you to do so many any validation if you're not a huge enterprise organization.

Basically, I had the doctor Pollard a couple of weeks ago. I couldn't figure out what was happening.

No, okay, I didn't even know they had to pull limit.

Yeah, they just yeah number something.

Yeah, it's funny story about that. So they're also doing some checks to make sure that you're not did those things that you're not doing.

That does attack on them.

So let me give you a story about you know what they do. The name of the company because they're actually talking about it by myself. So there's a company called Data Dog and I don't know if you're familiar with them, and data Dog.

Yeah, so they have a configuration the kubernetties.

And part of the part of the configuration was that you always need you you always need to pull a new image when the application is going out when it's deployed, right in image pull policy, which means that you need to oways pull it.

And they have like only.

Three land addresses, so it's pretty IP addresses, and they have all the images hosted somewhere. And someone made a mistake like de velotals of making mistakes, and it was actually a buggy code that got deployed with Kubernetes.

So what's happening.

What's happened is that it's got deployed, so it's trying to push the it's trying to pull the image. The code is not compiling correctly, so communities is noticing that something is not correct.

It's killing it.

But then it's actually raising any one because this is what kubernet is doing. But do it like one thousand times, ten thousand times, one other thousand times. This is what kubernet is doing, and doing it from three IP addresses to the same place. And if the vendor thought that they're getting a DIDOS attack, so they blocked and this is actually.

Very similar happened to me last week. Yeah, yeahs of times, but it was enough. Yeah.

So I I think it's a really good example of a misconfiguration that is actually passing validation because it would pass schema validation, but it's actually have policy that you want to make sure that you're not always pulling the latest image because then you can digdle something by accident. So this is something that will be checked, but it's it's Cobnetti's valid, but it's not valid.

Yeah.

I think I need to have an alert and that validator now instead of having a poll policy of always just have on whichever one it is not present or something. Yeah, I really need that.

Exactly.

I remember reading a few weeks ago about a Kubernetes manifest linter that would look for things like that. It would look for pole policies, it would look for do you have resource requests that are insane?

Are you asking for six thousand CPUs something like that?

This, this tool doesn't do any of that, I don't think, right, But do you use one and can you recommend one that that does similar stuff?

So this is a guy that is doing it's also yeah, again we are.

Not with heuristics in some cases.

Right, Yeah, you can also create like customers.

You can say that you can say, like, for example, that I want to make sure that there is a liveness prop and the value of the like I want to make sure that the entry point is always the slash else for example something, or you can make sure there's a cip you limit and it's always said to something like that.

Actually, you can do a lot of cool stuff.

You can say, like for staging, I want to make sure that the cip you limit is free, but for production, the if you limit can be six. So we can also mix them up and you can say I want to run this specific policy for this environment. Again, it's not something new. There are other tools that are doing that. I don't think that we created something that is unique. I think what is unique about our approach is that we're doing it simple, or we're doing it in a

And you have a big headed again.

So this is the project we are trying to take like we're trying to take it all off of your hands. You don't need to build all those integrations, don't need to glue them. Again, We're not doing something new. You can also do it. You can always do it with Jaq. You can also gluing by. You can also do it by yourself, but we will do it in an easy way for you. So you prefer to use the tree

It's fun fun.

Exactly, It's a kubernet It'll be fun, they said, exactly. I'm interested in asking a question is completely unrelated to this. In your introduction, you said that you're a leader or founder or something of Hub Users group, the largest in the world. Tell me a little bit about that.

What do you do?

I mean, I'm part of the Go users group, or we don't call ourselves a users group. We call ourselves a meetup group. That's the new version of users group right here in Amsterdam, and we just get around and get together and talk about ghost Tell me what you do with a GitHub users group?

Yeah, So basically like this article, it came from my own thing, and the paint was that I wanted to discuss someone about some features that get up ad and I tried to look with among my friends, like where do we have like guitup meetups that I can ask this question? And the answer was nowhere. So I said like, okay, that's cool, but I love get up. I'm using git up, and I'm sure that a lot of the veaples love git up and using it.

So let's do a itt up about geitub.

So this is how its gets started, and it's actually a user group because it's led by the community. I'm not working at Guitab, I'm not working at Microsoft, they're not paying me in any way.

I'm just doing it on moren free time.

So this is why it's called user group, and it's actually was surprisingly growing by itself because the first meetup was among one of the twenty people that register, and the last meetup that we did was eight hundred people registered.

So because of the numbers that were, where do you ask all those people?

Yeah, so the other numbers it's on nine, so it's usually it's on nine. We don't have a big place to hold so many people. And also there's like a benchmark that you know that if you have eight hundred people that are registering, not all will come. It's only thirty percent usually, so that's fine. But again it's a lot of beer and a lot of pizza to bring to Amita.

Okidding, Well, that's great, congratulations on that.

I mean, it's always fun to be part of a community like that and to get so much enthusiasm about whether you decided to start. I know that this has to feel good or maybe overwhelming or both.

Another fun fact, actually, my co founder that was also the on episod number seventy six again or about the cross reference here, is actually leading the local AWS community, which is also the biggest one in the world.

So it's a.

Little bit of a fight because it got acquired by Microsoft. So I'm like on this side easy the data by West side, and we are working in the same company, but we're still good friends and we love each other.

The group, because they're kind of Google related, that would be a nice little trifector.

So they started, they started from Google, but right more they're standing by themselves. Yeah, it's like part of the CNCF and organizations, so it's like Google studied. But I think it was really nice that they say, like, Okay, we realized that it's something that is bigger than Google and we want the community to enjoy it. So hey, CNCF, take this wonderful child and pa raise it for us the world.

Is there eight w us in Israel?

Like local locally you mean like like salvers, like physical service?

Yeah they do, they have like an office? Do they do they have like the physical presence?

Oh okay, So we have R and D and in Israel for and right now they're actually building like we that data.

Something centers in Israel.

So we're also going to have the computers themselves, like the machines on Israel Land.

We don't have it, so it will be all it will be holy service, I guess.

And there's something you said, Julian you said you don't have a w S, right, we don't.

Have a WUS like locally in the Middle East. So in the GCC, although they might be in Bob right now, I'm not sure, but within Uee and Doha we only have a JURE, which is a problem for me in getting local clients because I don't want to have to learn a lot of things, like I'm kind of lazy and a w US is enough, all right. It has a lot of things that I up with and that could be another story for another time, but like, yeah, for real, I don't want to move on to another

Yeah, there are a lot a lot of the centers in Israel we also have in the like there are a lot of the companies that Valenti is in here and because we have a lot of people that a lot of developers, a lot of qualified people to do that. The only thing that we still don't have is like the cloud provided themselves the mid local missions.

But like I said, it's going to be changed.

I know that Google is going not Google, but as she was going to open and a w is a going to open. We are using a west Virginia at aws.

By the way, everybody's using west Virginia cool. Anything else you don't want to talk about.

No, I think you got it all covered. So just to summarize it all, you should all validated, kubernet is filed. You should all do it as soon as possible. If it's possible to do it locally, to do it then if it's not possible, at least do it in YOURCI and I give some tips about how to do it. You can do it with the different tools that we mentioned. You can do it with the tree, but you can also do it with the other open source tools. You can do it with native tools with like you cattle.

Whatever you choose that's.

It, right on.

Yep, we will put your contact info in the show notes and then the last thing for us to do. Here are our picks for the show. Jonathan, you're excited. Do you want to go first?

Sure, of course, bring it on.

I'm reading, or actually listening to an audiobook that I think is amazing. I usually read boring stuff like O'Reilly books about Kubernetes and helm charts and stuff like that, but I decided to branch out a little bit, and I'm reading this Sid Meyers memoir, which is still nerdy because he's a nerd, but it's so fun and he talks about game design and how he invented these games that he made for those who aren't familiar. Everybody's familiar, right,

Oh that's super cool.

Yeah, I played Civilization from way back in the day, Like what was the first day first version it was on Microsoft Doss. I think it it was either version one, it might have been two. I want to say it was version one.

I think I starned with two, and I played like the sixteen different expansions for version two, and then I think I played every version since such a great game.

Civilization is good, it's safe. I mean, probably more of my husband in sanity when I was on bed rest with my oldest because I had something to like obsess over besides kind of bossing him around.

So that's that's my story, Jillian, You've got pick for us.

I do. So. I've been on a quest to go and clean up a lot of my terraform recipes and release them publicly out into the wild. And I found it really good template for doing that from this group called cloud Posse. It's I think it's spelled pretty much

That's awesome.

Yeah, make files, make files and read me I think might be two of the hardest problems and software engineering.

I still haven't given up. It's really it's becoming like a cultural age gap kind of problem for me. When I talk to new developers, I'm like, it's all in the make file. It's like, it's there, right, And I'm like, what's the make file, especially if they've been using like node and they're used to the package dot Jason, and then I'm like, what's a make file?

Like?

Sit down, you need to talk about this.

Sit down in that chair, we're gonna talk.

That's right, That is it?

Yeah, have you got a pick for us?

I didn't know that that is one. Sorry, I didn't know. You just make my own walk.

That's quite all right, I've got one.

And it's funny because I've heard about this for quite a while, and I was like, yeah, yeah, yeah, whatever, it's fine, and it's a screen protector for my iPad. But it's from paper Like, and it's as you might have guessed, it's very paper like because one of the things with using my my iPad and the Apple pencil

Is it iPad specific I will work on any tablet that you use with the stilist. That's a great question. I don't know.

I only looked for the iPad version. I would imagine that they've got it for pretty much any tablet. Yeah, because it's just, I mean, it's just it looks just like a screen protector you know that you buy for your phone or any tablet. There's nothing significant about it, but the texture of it feels like paper. So props to their marketing team for naming the product as well. All Right, I think that's it. We've got a wrap. Thank you everyone for listening. Yeah, thank you for joining us.