Yak Shaving, Part 2, Also Live!

Hey Ben.

Hey Matt.

So we were looking at the problem with our SSL certificate for uh, twoscompliment.org. In other words, you know, we wanted to be able to like host from just twoscompliment.org, not www.twoscompliment.org, which sounds straightforward. And through the miracle of podcasting, we recorded that many weeks ago. Our, our sort of, uh, attempts to fix it, but we never quite got there. And so I figure we should probably finish the job, try and get it so that our website's actually working, and, uh, everyone can laugh at how much we've forgotten between what may be back to back episodes as far as our listener is concerned. But what, for us, a month has passed.

Mm-hmm.

Well, we've got half an hour or so. Let's give it a go.

Let's see how far we can get at least.

So we had a whole bunch of Terraform Me stuff was how we left

It. Yeah. So I have, so right now, if I remember this correctly, our plan was to create an AWS Route 53 domain. And then change the domain to use, uh, like the wild card certificates?

I think so, yeah. We could use a Wild Cert. Wild Card cert, or at least a cert that has multiple hosts listed, one of which could be a wild card, but it could just have dub dub dub and the no domain, which I believe is what Compiler Explorer does. But I can't remember, I think, I think actually Compiler Explorer has like empty and star dot godbolt org or whatever.

So I have right now, so looking at this, so we had kind of terraformed some of this up before. Um, and right now there is a certificate that we have for www.twocomplement.org. And there's a little TODO here that should be, it says should be just twocompliment.org. And then I have another little TODO here that says Subject alternative names equals, and then square brackets quote star dot twoscompliment.org.

Right.

Um, and I have a little bit of four each magic in the Route 53 record that I think attempts to create a record for each of the things that it sees in the certificate. And I have this, I think because I have done this once before, and gotten this to work, and I copy pasta-ed some of that in here, but now I have zero memory of how it

Will all fitted into together

how we even got here, let alone how it works. So this is what happens when you put things down for a month and then,

And then don't pick him up again. Right,

Right. Uh, let me go check. I have a project on my GitHub that I think this maybe came from. So let me go see if that is even remotely true.

And if so, we can crip from that.

Yeah. And I guess I should try tastypenny.com. And that does work. And it is secure. So I have done this on that site. Yeah. My recipe tracking website called tastypenny. I have

Tasty Penny?

Tasty Penny. Yeah. I don't even know where is that thing. Yeah. You know, it's like all recipe websites are terrible

Because

It's all like,

They're not really recipes said websites. They're advertising websites that were trying to put as many adverts between the obvious thing you want, which is the damn recipe.

Right. And try to, you know, get enough SEO from the text that they're putting on there about like, oh, I traveled to Paris three years ago and I had this wonderful, you know, whatever. Yeah. But yeah, I can't even find, am I like not logged in? Oh, I'm logged in as, yeah. Okay. That's what's going on there. I'm not,

Oh, you on, I can't see

My right now, repositories. Oh, no,

I see you're logged to GitHub.

Right. I'm trying to find where this stuff came from. So I'm going to GitHub, but I'm logged in as a different user, so I can't see my press.

Whoa. You have more than one user.

I do. I have an aquatic user for my official Aquatic things, and I

Have Oh, I see. Oh, I,

Me, which is less official.

You're less official. I you're not the official Ben Rady.

I am not the official Ben Rady. I am the, I'm just the casual Ben Rady.

I see

. Um, okay. So yes. Tastypenny. Uh, here's some Terraform. It says site. And Yes, I think that is exactly where that came from because that looks very similar except some commented out stuff.

Uh, before we get too excited, if I go to Tastypenny, or if I could type Tastypenny. dot org? com?

Com.

Com. No dub dub dub. It is working. No ads, no junk, just Tasty apparently is the uh is the little, uh, byline for this and. Connection is secure, it says, and certificate is valid. And the co common name is tasty penny.com. And there you go. And looking at the, yeah, you just got a certificate and it only mentions tastypenny.com in this certificate. Now I'm gonna go to www dot tasty penny.com and connection is Secure Cookies, all the things. Maybe it redirected me then

It might have.

I see. But obviously in the interim, it, it was, it served up something which did not upset my browser. Yeah. In terms

Of you could curl it if you wanted to know for sure.

I certainly could, but yeah. That's awesome. Alright, so this is going to be a good thing to crib from because it works. That's what I'm checking is like, before we get all excited and changing it, let's just see that like we're heading the right way. Uhhuh and I will curl it actually.

While we're going. Yeah. So yeah, when I do a curl dash v I see server certificate, subject, see on Tastypenny start date, expire date, which is uh, in May. Okay. We'll keep that in mind.

Picking up rocks

Subject alt name host www tasty penny.com max. I see certs star tasty penny.com.

Perfect. Okay. So that's a good

Sure. Amazon

Analog then of that. And we know what to look for when we, when we do it for twos compliment.

Yeah. So, you know, the question with this is how do you fix the airplane while it's in the middle of the air? And, uh, I would hate to, uh, you know, apply some terraform change here that all of a sudden makes our podcast disappear for, you know, 24 hours or however long the DNS is poisoned or whatever it might be. You know,

So how about this? Can we make a change straight away to change the TTL of the DNS down to like two days and just apply exactly as is, but with a really low ttl, which means that already, or like two hours, which means that we're starting to promote the idea that we're gonna screw this up and we wanna be able to undo it. Right. You know? Exactly. Yes. A good friend of mine once told me that if you can't test it properly, then at least make it cheap to roll back.

Oh, yeah. Okay. That sounds like that guy was making **** up as he went along.

Right. I think so. Right. So we might be, uh, I mean DNS is it's own mysteries

The question is, I don't think that currently AWS is the name server for twos compliment. I'm using the other provider.

That means that we can make all the changes we like here and just use NS lookup with the server being, or host or dig or whatever the cool kids use these days, um, and test that it's doing the right thing.

Right.

So right now, I, how

Would we, how would we confirm that that is true?

I'm gonna do NS lookup and I'm gonna do set type equals any and I'm gonna do twos and sorry for my offensively loud keyboard twoscompliment.org. And it tells me non authoritative answer name server is ns2.hover.com and then,

Yeah,

That's blah, blah, blah, blah. And then address is 2 16, 40, 34, 41, whatever. Yeah. Okay. Yeah. Now, beautiful. If I were to set the server to be, do have you have, do you happen to have a a, a AWS DNS IP there?

Uh, let me go see if I can do that. One moment please.

And meanwhile, I'm looking at the Tasty Penny website going, this looks great.

I wanna, I, I, I have some updates I wanna make to it. Uh, I want to make it a little bit more tablet friendly, cuz it's not right now, but it, you know, it has some good recipes on it. Uh, yeah, I wanna sign into the console. I don't know if you've got this thing recently where I've finally had to separate my amazon.com, you know, ordering

Oh shopping,

Shopping, shopping password and my AWS password. Yeah. Through a reminder of like, it used to be that Amazon was a bookstore.

Yeah, that's right. Yeah, it is pretty bonkers. At one stage, actually, I had a problem where, um, I enabled two factor authentication on one or other of the two and it affected the other, even though they were supposedly independent. I think there's still some link between the two. They're different account names now. And I, that was the one and only time I ever spoke to an, an Amazon person on the phone while they were trying to reset it out. Oh, this is an interesting problem. I'm like, yes, yes it is. I can't log into either. And this is kind of panicking me right now.

Hmm. Uh, okay,

Well this padding. Yeah. Has that allowed you to find the IP address of

Yes, yes, yes. So I have the twos compliment name servers.

Yep.

There's four of them.

Anyone will do,

Let's go NS dash 60 eight.aws dns dash zero eight.com.

Wow. That's a beautiful thing. Oh, the, so it has an IPv6 address. There you are. That's crazy. Okay, so now I've just said server that, and I'm typing twoscompliment.org again. And now it tells me, uh, the name servers are, now I can see the other DNS uh, servers at Amazon, which are like NS 1, 1 50 and one 1600, all this kind of crazy things. And apparently it has an address of 202.251.192.68 is what it's resolving to, which is that the alias to the load balancer?

Yeah, it's a cloud front distribution.

Yeah. Okay. So I'm gonna, I, what I'm gonna do is I'm gonna look up, no, yeah, it doesn't, oh wait, it's refusing me. I want me do this on another one 1, 2 0 5, uh, 2 51 1 2 2 68.

Uh

Oh. That says NS 68 aws D n I mean, it could be the same IP addresses for all I know. So yeah, we need to look up what the cloud front, um, distribution is set to, just to see if just, I mean obviously this is just us testing the water here, like this should, should all work out. But while you do that, um, what this means is, yeah. CloudFront is essentially a caching proxy in front of all of the, um, aws, uh, infrastructure. And, um, when one creates one, one tells it where to get the information from that it's gonna be proxying and, and edge caching and it gives you a sort of a unique, uh, distribution name and then that maps to an IP address or a DNS that you then use to point your web services at. And then whatever you land on knows how to serve up from CloudFront, your web server, your web traffic, your web, whatever.

So I have arn, I have a distribution name, but I'm having a hard time finding,

What is the distribution?

An address? Oh, wait, no, maybe this is, maybe,

I think it is and I I think it might actually be a full FQDN

D n I mean the distribution name is a Okay. Yeah, I think I do. Yeah. Okay. Uh, this is, this is gonna be a little painful. You ready for this?

Okay, then. Alright. Right. Well maybe, yeah, go on. Is it as something or other?

It's, no, it's, it's a big long stream string of care characters.cloudfront.net.

Okay. So why, if you've got that in your console, why don't you copy that and just do host space that and see if you get an IP address that looks like this one and then that will sort of con confirm while you do that. I'm another copy, www.twoscompliment.org

Uh, I see 54 2 30 18 99. Okay. 54, 2 30 18 49, 54, 2 30 18, 82 54, 2 30, 18 69.

Got it. None of those matched because I've just realized that NS lookup was giving me a bad answer. It tried to connect and it got time out. And then what it's doing is it's just telling me all about the domain. There is no, i, there is no, uh, a, uh, record associated with twocomplement.org. So that's what we need to fix.

Okay.

There's also no a, there's no a record associated with dub dub dub dot two complement do org. Right.

Cuz isn't there not going to be, isn't it gonna be this like different kind? Oh, what are the, what is the name of that type of DNS record that's like, it's not specific to Amazon, but it's like,

Well, it, it's called sort of alias around. So like the, the, the underlying problem here is that there is no such thing really, as much as people would love there to be, there is no such thing in DNS as an, uh, a c name, which is what we really want for. The, um, uh, the, the naked domain. Like, so what you might want mm-hmm.

Yeah.

And so what typically happens is that DNS providers will have a product where they track the DNS entry for the cloud front end that you've got and they'll just keep periodically changing your A record. But Amazon natively supports this, so we should just be able to configure it. So I think we're just missing the configuration in the Terraform and an app, an application should just make this work here without affecting the real twoscompliment.org Cause it is still being served up by hover.com.

Right, right, right. So I'm actually looking at this now, and this, this makes sense to me, which is I've actually got some commented out stuff in this Terraform that does, I think exactly that. And that is what my Tastypenny Terraform does. And looking at my Tastypenny configuration in Route 53, I can see an a record there. Um, that is a very strange looking a record because the value of it is that big long list of characters cloudfront.net. Right? Well, not the same

One. Oh, that's interesting. Yeah.

Um, uh, and that is for, uh, the www one and for the sort of bare domain. Yeah,

That sounds,

So that to me seems like Amazon, you know, doing an a record, you know, trick.

Trick behind the scenes. Let me, I'm gonna have a very quick look at how I did this for some other website that I'm, I'm involved in, uh, Route 53 tf, where the hell all this stuff? Uh oh. Yeah. I actually have modules for this because it's, uh, so awful that I have so many stupid things. Main tf uh, okay. I set a CName and the records are the, yeah. Something like Route 53 address a FQDN. So it's kind of looking up somewhere else. I'm trying, this is obviously makes for great radio, um, uh, zone the alias name. Yeah. Okay. It looks like it's an alias that I'm setting. So I do, for both the, the A record and the AAAA record, I have, um, an alias stanza inside of the Terraform itself. So it's not an address record, even though it could be. And it has a name, a zone ID and some other bits and pieces in it. And I dunno if that corresponds to the thing that you are looking at now.

Yeah, I think that is, I think we're looking at the same thing here.

So I've got, yeah, alias name equals, and then I've got a variable which holds the CloudFront distribution domain name, and then another thing that's CloudFront distribution dot hosted zone id. And that essentially configures the A and the AAAA for the top level name, which is, in my case, you know, godbolt.org or godbo.lt or compiler explorer.com for all the times it's instantiated, which is like the four each that you've got. But I think we only need one of these. So you could probably just write it out longhand right now.

Yeah. Well I, I think this would actually just work if the certificate was the, if I switched, so kind of parsing through this now and having some vague memory of what we did here. Yeah. Um, I think the, this will all work if we can just, change this certificate to be a wild card certificate. So if I were to change that in the Terraform and then try to run it. What would it just replace the existing certificate with a wildcard certificate?

"just" I think so. I think so. I, I have some magic to do that too if needs be. So why don't we try, try that.

Let's give that a try. Okay.

What could go wrong? We could, I mean, right.

Well, in theory people could start getting certificate errors going to, to twoscompliment because I do think that this is the real certificate. This

One will be the real certificate. Yes. Right. The DNS can do whatever it likes, but we're about to tell CloudFront to use a different certificate when it's pretending to be us.

Yes. Which is probably why I stopped here.

I mean, Yolo.

Let's do it.

Did you make a new certificate? Actually, you already made a certificate.

Well, I, I was gonna, I mean, can Okay, wait a second. Stop. If I change this Terraform, it's not going to make a new certificate. I have to go and do it. Manuel-y?

Uh, I don't remember if you, I mean you can absolutely have certificates created in Terraform two. I don't know if Did, did we do that last time?

Okay, well let's do this, let's start by making the change in the Terraform and doing a terraform plan and seeing what Terraform says.

What the heck it thinks. Yes. Always, always a good start. Where are we now? Is what am I, is what I have on my computer an accurate representation of what the cloud provider thinks I've got.

Right. Right. Well, I mean, so I did this once and it said it was up to date, but I'm gonna change it. And now we're gonna do a plan again, and then we're gonna see what Terraform says about what it feels like it wants to change. And I'm gonna make this look very much like the existing one that I have for my recipe project. Right.

Okay. And I found the certificate stanza that I have for my site so we can steal from if needs be.

Okay. So I'm gonna do Terraform plan.

Yep. What is it saying? It say refreshing.

It says three to add, one to change, two to destroy. And so it is going to Yeah, say AWS acm certificate twos compliment.org must be replaced.

Awesome. Okay. Cuz you've changed the subject alternative names in there, right? Uh, I

Think it's, uh, I changed the subject alternative names and I also changed the domain name from www.twoscomplement.org to twoscomplement.org.

Perfect. Perfect.

Uh, and then it says AWS CloudFront distribution. S3 distribution will be updated in place. And then it says, uh, AWS route 53 twoscompliment.org bracket star twoscompliment.org will be created. And then another aw, uh, route 53 record for twos compliment.org will be

Created. That sounds good to me. Let's do it. What could go wrong? Well, let's get a list.

Many things,

. All right.

All right. So do I have an applied, do I have a, oh, I do have a Terraform apply. All right. Firing the rockets.

Firing the rockets

Rockets. Um, if we wanted to troll our audience, we should cut off the audio in the middle of this supply

Yeah.

Doesn't work like that.

Well, ironically, you're stitched then for me, so I'm like, oh, he's joking about the connection going down, and then I'm like, you froze on my screen.

Oh, man. Which was

Epic trolling in its own right.

All right. It says destroying still destroying, still destroying. Still

Destroying. It's destroying

Everything. Oh. And we got an error.

All right. Is there some crate before destroy thing that I've, I've got in mind?

No, this is a, uh, what does it say? Access denied not authorized to perform ACM request certificate. So this is where we go into the IAM console and we give this service user that we're running as has a whole bunch of permissions that shouldn't probably have, and then we dial 'em back later.

Far too clever. I just

Have, cause I don't, I don't actually myself manage the, I, no, I don't know. I don't manage the IAM

Here.

Okay. So where is this user? Tastypenny. And, uh, yeah, we're gonna attach a permission and this is gonna be, um, what is the name of this service? Certificate? Something AWS certificate. I think it's this one. I don't even know. Who knows? AWS Certificate Manager Private.

Yeah. You, this is outside of my purview of understanding. Well, that's not even the right word.

We'll try this one. Yeah. And see what happens. And if this doesn't work, then we'll remove that. We'll take that out. You know, if it ain't fixed, don't break it.

If it, yeah, if it don't, don't leave it broken. More broken.

Yeah. That's

Like's, yeah. The programming by coincidence thing. I think we've talked, have we talked about that before?

Um, maybe, maybe not.

Yeah, maybe not. Maybe

We have not. Yeah, so that didn't do it. So I'm removing the policy because that did not fix the problem, so I don't want to create a whole other problem by putting something in there that wasn't in there before. Uh, but I AWS certificate, so this guy should have this already. ACM.

Association of Computer Machinists. Yeah. No, not that. What,

What? Oh, I guess I can go and look at this actually and see what it's, yeah. This, this user. I th I thought

Are, are you the right user though? Is that,

Oh, it's a different user. I'm an idiot. I'm looking at the Tastypenny user, which

Clearly does work, which

Already works. Yeah. Like this. I did this already. This is the one that works. I guess I should have thought of that before. It's like you have a user that does this. Go look at what they do. I'm a doofus. I think I was maybe thrown off by the, uh, fact that, uh, our, the user that I have for this has the original podcast name. Can we talk about the original podcast?

I don't think we talked about, oh my golly. This is all these things. Yeah. Programming by. We, I should be taking notes.

, uh, all right. Certificate.

All right. Certificate. Give me all your certificates Are belong to us. Oh, I've just gotten

Certificate manager.

I dunno if this is, this is certainly completely off topic, but I've just been given the okay to push an update to compile Explorer, which I will do in the background of this. So the continued tapping noises will be me pushing a kind of cool thing to compiler Explorer.

Okay. We're creating, we're creating a certificate. Alright.

Oh, uh,

That's, that's a good sign.

I'm pushing Compiler Explorer 6 7 25 2 production from the staging environment, unrelated to this podcast. But, you know, we're all tap, we're both tapping away our keyboards if we've got a filter to the air with talking or some description.

So, yeah. So it's interesting to talk about how we would do this if this were not just our hobby podcast. Right. So cuz right now we are literally testing this in production, right.

I've seen, we've all seen the meme, the, the most interesting man in the world meme, you know, with him, with his little beer going, you know, I don't often do testing, but when I do, I do it in production.

I do it, I do it in production. And that

That's not our, that's not our, our our mo in our day job. So if anyone's thinking that this is the kind of cowboy activity that we would do, if it was anything other than you and me chatting

How, how would we do this if it was, well, so obviously you want to have a separate environment for testing this out, but the trick with creating that separate environment is how do you know that your separate environment is a copy of the state of the environment that you want to change for real.

Right. Right.

Um, which has the additional problem of it is it's gonna take you some time to make these changes and in a large enough organization or in a large enough project, that means that the environment, the production environment may change while you are working on making the changes, right? Yes. So you might be able to make a copy of your production environment as it stands right now. And then make some changes to it, test those changes out. And while you're doing that work, someone else might be doing the same thing and making other infrastructure changes to the main environment. So when you finish that, you need a mechanism for basically reapplying the changes that you made on top. It's almost like a, like a fast forward in git right? Yeah. Where it's like, yeah, yeah, yeah. You need to reapply the changes that you made on top of the environment as it exists now, not as it existed when you started working on the, on the new thing that you wanted to add. Yeah, yeah, yeah. Right. Yeah. Um, so I feel like the only way to even have a hope of being able to do this is to just automate everything. Infrastructure is code style with Terraform. Like, I, I feel like,

And have the only thing that pushes any of this stuff to be the main branch of your GitHub repo so that you've kind of post hoc, already merged everything in at the point of where things are applied. Um, you kinda get a merge commit queue at that point, right? The only thing that's really making changes to your production deployment is the, the, the, the head of the line where all of the, the intermediate branches have to definitionally have been merged in. Otherwise it goes, oh, I'm rejecting you because you know, you're not at the latest, you know, oh, I have to get it again or whatever. That kind of feel or are you Yeah,

No. Yeah. I, I think it, I think it is that and, and then being able to sort of rebuild your test environments based on changes that are, have been actually deployed. So being able to either tear them down and build them again. And reapply the new things that you did or merge a change in, in a way that's realistic. Like, like, you know, it's, it's probably like the order of operations, uh, potentially can result in this in the same environment where it's like I had some environment and then I applied someone else's change and then I applied my own change. Uh, that is probably, that is representative of what is gonna happen in the main environment when you merge your change. Flipping them might not, right? Like if you apply yours first and then there's, like, you might get the same thing hopefully if Terraform works the way that it says on the tin. Uh, but you might not, right? Yeah. So you have to like, think about like how that's all gonna get applied. Uh, so speaking of Terraform, that doesn't work. Uh,

Okay. Well,

So I don't know if we destroy our other certificate and made a new one or what just happened

To here. I think you do up arrow return and see what it does the second time because some of these things have disgusting. Like, oh, it takes a while at the back end of,

Of,

Of, um, which is not ideal.

Especially a certificate, right? Yeah. Um, all right. I'm gonna go look at the cloud front distribution.

Yeah, that's a good idea. Um,

And see what state it's in right now. Uh, it says it's enabled, uh, can you curl the site real fast and just see if it returns anything?

I can certainly curl it

If it gives you some sort of weird certificate error.

Um, Oh, hang on a second. Dub dub, dub dub twoscomplement.org importantly. Cause that's exactly what we're trying to fix,

Right?

You see, this is why it's a problem for me. This is why we have to fix it. Cause I I'm too lazy to type dub dub dumb or even say it properly. Yeah, no, it's working fine still.

Okay. Whatever it is. Yeah, it probably created the new certificate and was trying to flip the, uh,

The, the, the cloud front to it

Over. It was like, no,

I, you've got the console open too, so you can actually have a look in the ACM certificate thingamajig and see if it's there or not, or

Oh yeah. Good call.

I know. So we've deliberately not shared screen so that I have to ask Ben what he's seeing so that, that you dear listener can actually sort of hopefully follow along. I dunno how much anyone will be able to follow on what we're

Doing here. Yeah, I see. Okay. So, um, yes. So I see four certificates in here. Uh, two of them are twos compliment ones, one is the www one that is, uh, issued and in use and eligible for renewal. And another one is, uh, without the www its status is pending validation. Ah.

So,

So we may have to wait. Uh,

There's usually a DNS validation. That's how these things. Did you have, what type validation did you have? Is it, I mean, this is, it could be that it's, you might have an email right now because it's like, Hey, are you really sure this is your certificate?

Oh, interesting.

Uh, mine set up for dns, which I think because Route 50, whatever monkey, uh, uh, is in cahoots with itself, it can basically set its own DNS records and reque. Right? Oh. But there's the problem. Now we've got two, now we have two problems. We can't, we won't be able to use DNS validation because you honestly haven't flipped the flag yet for the real DNS provider to be Amazon.

Yeah. I could copy those things over into the other one though. Right?

You certainly could. If it tells you what the, uh, challenge is that it's put in the dns, then you can put them

In. I mean, I could go if it, if it added it automatically, I could go look. Right. Go look around 53 and be like, what did you add to this thing?

Yeah, yeah, that's

True. And just copy those over. Uh, but yes, I agree with your assessment of the situation here.

Yeah. Which may have been, I, this rings a bell from the last time we did this and like, hey, yeah, this thing might take a while

Yeah, yeah, yeah. Um, so I don't see any new records.

Maybe it's not set up to do it that way. So I mean, if you look at the acm, it's the certificate. Does it say why or how to au to do that thingamajig?

Um, it says pending validation, renewal status, number of additional names.

And you don't have an email or something. I can't remember how this works if it's not set that way.

Not that I see.

Just checking my email. Cause some of those addresses you put a little forwarder on. I don't know that it's

Yeah. Nothing in my, uh

Oh, right.

Spam folder real fast, just to make sure.

Yeah, I can't help. I'm gonna go and find my, my certificates

No

Certificate manager. Oh, of course. I need to log back in again.

How did I do this for,

For your magic penny.

Yeah.

This, I'm just looking at mine and I can see Yeah. In use CS renewal. Elig durability. Right. Okay. Um, so I can see, oh no, that's, yeah, I can see that if I, I've gone to one of my certificates and I can see that it has in the sort of more information inside the console itself, under the acm, uh, it's got a list of domains and it tells me status and renewal. It's just type and then it's got cname, name and cname value. And those are the two things that need to be put into the route rty thing. Oh. And there's even a button that says create records in Route 53, but you can click

Oh,

Well, but obviously you don't wanna do that because we don't necessarily

Yeah, yeah. Right. Okay. Let's, I'm gonna go into my other registrar, dudad, right. And I'm gonna go to choose compliment.org. Why is there oh one

be very careful. Very careful.

Yes. But I gotta make sure I, I mean, it's not gonna hurt anything if I do the wrong one, but like what

Fewer things

If that, uh, so dns, and then we're gonna add a record and it's gonna be a cname record,

Which you can copy paste from, thankfully from the other thing.

Yep. Yep. And then that's

Gonna be how, honestly, how much of software engineering or administration is goes through the clipboard. I mean, it's just,

Oh my God. So much. So very much. And I'm gonna set the TTL to five minutes.

Wonderful.

Gonna. Add this record, and then I'm gonna do this same thing again for the wild card. Yep.

Yeah, you've got the two I, I can see for each of my domains, I've got two Thingies.

Mm-hmm. And then,

And then of course we have to hope that it notices this within.

Yeah. All right. So yeah, I've got set two of them set here. Um, and it's, it's probably a good sign that I ha actually had another one for the www certificate. Yeah. That is in here. I can see it.

Right. Okay. So,

So now there's actually three, right? Right.

But these are all like, interim. So like, just to sort of recap in case that we're, we're, we are trying to prove to Amazon that we own that domain name. And one of the many ways that we can prove that is to make a change to the dns records with some magical things that they've given us. These are the C name records that, that we've just been talking about. Mm-hmm.

Mm-hmm.

.

Okay. Exactly. Is there a way to poke the AWS certificate manager and say, Hey, can you,

Can you take another look

Now? Range to target one ping only, please. Yeah,

Yeah, yeah. Come on. One ping only. That's a good,

Uh,

That's amazing.

I don't know if there is, maybe, maybe I can do this here. I can delete it. I don't think I want to do that. Uh, request.

Yeah. Does that maybe gonna make a

New one? I can say that's probably gonna make it, well, this is where we manage x free events. Yeah. This might be we just wait, you know, 10 minutes

For, so

We just wait.

Well, compiler explorer is 67% through doing an update very excitingly in another window. All Mm-hmm.

Right. Mm-hmm.

So that's,

Uh, oh man.

So then we were talking, right, two things. We talked about one, obviously we just, we, we sort of briefly mentioned was the idea that in our day job, the way that we do this is that the CI build in main applies the production configuration. And so it's been through all the testing and there's not like the two people fighting over two independent things, uh, changes along the way because you always are seeing the union of whatever has been merged into trunk

Mm-hmm.

Then how do you test it? How do you test a separate like thing? How would we, um, so in Compiler Explorer, I have some very hard coded staging and beta, or beta just to con uh, de confuse people. Honestly, I've had this conversation so many times with Americans, they're like, what beta? And they're like, thinking like egg beaters or like uhhuh,

Yeah. I, I don't remember if we've talked about this on the podcast or not, but we lately have been doing a thing, uh, with a, a data warehouse project that I'm working on where the branch in GitHub represents an environment. So we don't have a production environment. We have a main environment because we have a main branch,

Because the main branch is that Right? It's not special case in any way.

It's not special case. It's, there's like a couple of additional protections for deleting things.

Got it.

And that's it.

But other than that

That, you have to, other than that, it's identical to every other branch and identical to every other environment. And so when you create a new branch, it, you know, says, oh, this environment doesn't exist. I guess I need to apply this Terraform, I apply the Terraform every time. So Terraform just has more, more work to do this time. Uh, and it, you know, spins up all of the infrastructure that this project requires, and it's doing that obviously from a fork of the Terraform file that was just in the main branch.

Got it.

And is therefore a copy of the infrastructure that is running in the main and environment. So you wind up making an exact copy of whatever the environment was at that time. Right? Right. Um, and so that will all get created. It will then automatically deploy, uh, to that environment. And now you have a completely separate running copy of that system.

Exactly. As we, or at least been doing right now, except that because we don't have this set up, we are experimenting directly in prod. Right?

Exactly right. Exactly right. And so once you are confident that your changes work and that all your software works with any other infrastructure changes that you have made, you can at automatically merge those things back into the main branch. So your infrastructure changes and your software changes that may be interdependent on each other, all get merged into the main environment at the same time, uh, the same sort of Terraform application process that you used in your branch then gets applied to the main branch, your new software version gets deployed, and if everything goes according to plan, uh, now you've updated your environment while, while doing so in a way that gave you high confidence that the changes that you were making were actually going to work before you tried to do them for real.

Right. Right. And presumably, like we've discussed before, if it, if it doesn't at that moment in time, the hope is you could just revert that commit to main and it goes back to everything before. As long as Terraform does it, its job. And as you know, if anyone from HashiCorp is listening, never, I don't distrust it in any way. Uh, it, it's pretty reliable. So you can almost bet the farm on, on it doing the right thing most of the time.

Yeah. Yeah. There are gonna be some situations in which you can't figure out some path to go from wherever you were, wherever you are. But really, I would say 99 times out of a hundred, uh, it does exactly what you, it would expect it to do. Right. So if you revert that change in the, in the main environment, it's gonna then have a different Terraform configuration and then Terraforms gonna try to change that configuration. Um, you obviously have to be careful of things, and this is why we have a few individual protections in place. If you were to say add a, um, an S3 bucket or add a data store, add some other thing, roll that into production, write some data to that data store, and then realize that you had another unrelated problem, if you were to roll that back, it might, it's going to delete your data store. Right. By default. Yeah. And so you want some additional protections in there to say like, Hey, if you ever try to do this, just don't

Sounds awesome. Uh, from my own personal experience, the trickiest part of this is when you start doing refactoring in Terraform and you wanna like say, well, I do have 10 running e c two instances, but they've got terrible names in the Terraform. And I wanna rename them in Terraform, which means I have to do this unfortunate two-stage thing where I changed the name and I don't wanna delete them and recreate them. I want them to be this. And there's ways and means inside terraform of like using state to actually say, okay, I'm renaming this thing in the actual, uh, state. And if you go full automated, you, you don't have the little breathing room to do that. Where I'm like, I have to kind of literally call around people and say, okay, I'm doing some like surgery on Terraform, I'm gonna rename this thing, which means I have to rename it in the backing store, which is a Terraform command, and then I'm gonna change the text file, and then I'm gonna do Terraform plan. Then it should say no changes needed. I'm like, good, because I didn't really change anything. Right. So I dunno if you've had any, uh, experiences with that stuff yet, or do you just say, I

Haven't had to go through that process yet. Um, right. Part of it is because, and, and I think this also is sort of related to another, uh, potential trade off with this approach that I'm talking about, is that your branches can get very expensive. Yeah. Right? Like if you have lots of infrastructure that has like a per hour cost to it,

Right. Load balancers, for example. Exactly.

Uh, then, you know, running a branch can be, can be very expensive. Right. And so one sort of side effect that, that I have kind of seen or felt working on this project is that it leans, it, it, it, it leans me toward using more like serverless things and things that can basically scale from zero.

Scale to zero. Right. So Yeah. Yeah. If you have like auto scaling groups, you say, well, they start out zero and the first request that comes in, unfortunately it's gonna be delayed, but that's fine for this.

Right. Yeah, exactly.

Or, or as you say, lambda type things or Yeah,

Yeah. Lambda type thing. I mean, there's lots of them out there. Right. But it sort of has, has me using those things more because I know that, you know, we're gonna be creating a lot of these branches and we wanna be able to iterate and it's like, yeah, if you use them, you want to scale up to be able to test them. But you know, if you're not using some particular functionality in a branch because you're testing something else, you don't wanna pay for it. Yeah. Um, so, you know, for better or worse, it's, it's sort of like the architectural direction of this project has headed in, in that way. Um, just for cost reasons.

That's really interesting. As I say, like, well on the extreme end of like what compiler explorer does, I'm like deliberately sharing a whole bunch of things so that I don't pay the cost for the low balancers and the storage or whatever. And the other thing that we deliberately don't bifurcate is the storage of a whole bunch of stuff because we have, you know, three terabytes of crap and you know, there's no way I'm gonna keep deploying that to a new environment every time one gets spun up. And similarly, I wanna be able to create a short link in one domain and test that it still works on the old version or the new version and stuff like that. And that's sharing that tables behind the scenes. So there's some sort of edge cases with that. But I would also like to be able to say, no, I just want a whole new copy of the whole thing somewhere else so I can make a wholesale test.

Mm-hmm.

I suppose so.

And so what I want to copy data from one environment into another. Turns out we have a lot of great tools for that. Um, you already,

That's part of your mo Yeah,

Yeah. Yeah. So we sort of lucked into that, but otherwise it would be kind of painful. Like you'd either have to have a thing where you have, you know, like maybe read permissions into the main environment for many of the branch environments so that you can sort of test things out.

Right. And every time you do it, you're sort slightly eroding the nice guarantees that you had before about like the isolation of things and whatever. Right.

Exactly.

Sometimes you just, this is what I mean, this is what makes it engineering and not science or art. Right. It's like Right. There are trade offs all the way through this.

Right, right, right. And we have had one situation thus far on this project. It's been going for about six-ish months now, something like that. We've had a one situation where a change in a branch environment leaked over into the main environment. Oh. And this was because of this thing. We had some data in the main environment, uh, that was being reused for testing in the branch environment. And additionally we had a permission that was set incorrectly. Right. And what had happened was basically the, uh, system running in the test environment saw this main environment data and said, oh, I need to go disable this object, this thing, this resource. But it was the main resource. Right. Um, and it went in and it disabled it in the middle of the day. Right. Um, and so it shouldn't have had the permissions to do that, but, you know, permissions in, in AWS and in Terraform can be a little tricky to get Correct

As, as discussed today, you know. Exactly. It's not necessarily the easiest thing to get. Right.

Yeah. Yeah. It's not like you can write tests for those kinds of things, so you just have to sort of like, I

Don't know if they mean I know they,

Cause if there's a testing framework for AWS permissions? That would be kinda amazing.

AWS has a built in, um, permissions thing where you can run what if scenarios, but it's a very much as a service. It would be cool if there was a standalone thing. Oh yeah. I've That allowed you to sort of write these things where assert like given this environment and this mm-hmm. User assert that they would not succeed in deleting this file. That would be pretty Right. Pretty cool. Right. Maybe something exists.

Seen that, I guess you could, you could maybe do a thing where you like decorate parts of the AWS SDK and you say, run as if I had this policy. Right. And then you could like, try to do operations against a, basically like an, a non-existent environment and say like, you know, you don't have to give me the result, but just tell me what I would've been would I have been permissioned to perform this action

Order? The the real trick though is that it's so incredibly complicated. It's not like there is a policy.

Well, that's true.

You know, the user, the IAM role has a policy, the user has a policy, the machine you're running on has a policy. The then on the receiving end, like, oh, the, the bucket has a policy that grants anyone with like a name who's, you know, ends in a queue, they're fine. They can write to me. You can do literally anything Right. As well as the other way around. So, I mean, who knows?

Yeah, yeah.

Yeah. How is our certificate doing?

Uh, let's give it

One more check. Let's give it a go. Cause we're running out time here and we might, this might be a a, a ramly third part coming where we actually get it to work for reals.

Yep. Yep, yep, yep. Okay. Drum roll, uh, certificates. It says it's issued. Oh, let's try running the Terraform then. Terraform

That would this what a wonderful way to end if we actually, well I say end, we still got more work to do, right? Because we always have more work to do

Uhhuh

Okay.

So that's cool.

And then the, has it made the DNS change? Cause I, that's something I've still got open in the terminal is I'm still got DNS looking up to complement org to sort of see if there're Oh, I guess once the cloud point. Yeah. It has to be after the cloud front, um,

Stuff. Cause yeah, so I would expect the CloudFront distribution would use the new certificate, but I don't know. I'm trying to remember.

But you haven't put the alias in into the, the dns.

Yeah. And even if you, yeah, let's see here. Actually no, I think it might. I think it might. Okay. Let me go take a look here.

So I'm still not getting it on that and I'm talking directly to it, to the DNS. That should be reserving up these requests. Yeah. There's no caching going on, so I dub up.

I don't, I don't see it in the console. I'm hopeful that when this Terraform applies that it will actually

Got it. So at the moment it's modifying the cloud front a uh, um, thing. And presumably yes, because in the, um, in the new DNS records, you use a VAR that comes from the the CloudFront domain that is its unique name. It's probably depends upon it. So it's waiting for that to be applied before it does it, even though we know that it would be kind of okay. So, all right. Well we CloudFront takes a while.

Oh crap. No, that's still commented out. Okay, well

That's quick to apply though. So we can probably,

Yeah, I'm gonna have to, I'm gonna have to add that in there.

Yeah, that's all good. And compiler explorer is rolled out, which uh, is other good news. Um,

Okay. And then yeah, so that's applying.

Yeah. The cloudfront takes a while as it has to kind of get permit, uh, the Okay. From all of its geographically diverse, uh, regions before it says Alis gut.

My guess is that I'm also gonna have to add in a couple of these guys

Here. We're so close. We're so close. I'm actually gonna say to the people who I'm supposed to be now meeting that I'm not going to be there. Most of whom have said they can't make it anyway, so this is fine.

There is that

Should probably check on the other computer that I'm not being hassled or harangued. Long silence will be cut from the podcast during the edit.

Yeah, we can as we wait, we can do that.

We can, the magic will.

Okay. So that applied. So I'm gonna do one more plan for these other Route 53 changes and then I think,

I think we're at it and then

Getting close. No, I have an undeclared resource probably cuz I spelled twoscompliment wrong would be my guess.

I do that all the time.

Yeah. Uh, well, I put a.com instead of a.org. That'll do it. That'll do it.

That's not really a misspelling, is it? I mean, strictly speaking,

. No, it's just wrong.

Okay. What does the plan say? So good. The story so far that the CloudFront domain is using the new certificate, and now we are about to apply the DNS changes that will be still not used by the internet at large, but will be used by my console that is set up to use Amazon directly.

So yeah, we're, we're creating two, we're removing a, uh, a Route 53 entry and adding two more. Oh. Um, oh, because one of them, God, love it.

Oh, What happened?

I think I did a.com somewhere.

Oh, really?

Maybe it doesn't, hopefully after this runs, I just, I just named something.com. It wasn't actually like a domain name.

Oh, okay. Right. It was just resource name com. Like a variable name. Effectively, in, in, yeah. All right. Yes.

Okay. Well, so still running, but

Look really cool in the, in the edit because it'll just work first time. Mm-hmm. Every time. Mm-hmm.

. Yep. 60% of the time

works a hundred percent of the time.

Uhhuh.

All right. So we're ways in an application, uh,

Try to create record. Set a record, but it already exists.

Oh, did you manually make one before or, or have you duplicated it accidentally in the Terraform and Terraform hasn't noticed this mistake, which is

I think that is exactly what I did. Yeah,

That's, that's my Emma. Because like Terraform will go, this looks valid to me and it'll do the plan, and it said, this is what I'm gonna do. And then Amazon turns around and says, no, no, those are the same thing. You fool, you already got one of those.

Mm-hmm.

I told them we already got one

Two

Oh, I just went to just the naked twoscompliment org, and that has applied. I can see that it has lots of a records for all of the various different service. So that is excitesightful. We just need dub dub dub to be the same, which are you are working on, presumably.

Yes.

And see, this is one of those examples, incidentally, if you've already created one of those, um, things in the console, and you've got it in Terraform as well, which, then that's one of those things where you adopt an existing, you know, Terraform import, and that's harder to do in an automated environment, unfortunately. Mm-hmm.

Yeah. I really feel like you gotta be all Terraform or No, Terraform, you know what I mean? Yeah. Like living in the middle ground is just

That's true. But like, you know, you have legacy projects, for example, course, where you need a lot of adopting of what's there. And my, my usual trick is to, um, write a skeleton of my best guess as to what I think a, a resource that I already have looks like, and then import it. Mm-hmm.

Uh, it's saying try to create resource set twoscomplement.org a record, but it already exists.

That's because it does already exist. Now I can see dub dub dub twos com org is also those addresses. So that's good. Good from a It's working. Yeah. But not necessarily good from, it's, uh, it's gonna work each time we apply cuz it thinks there's something there that is,

This is almost certainly something that I'm just like copying wrong here. What in the great googly moogly

Sometimes when we say stupid things like this, it makes me worry. Uh, not worry. It makes me feel sad for you when you have to do the transcription of these, because the automatic stuff has got no hope with a lot of these words.

Yeah. True facts. Uh, so those two records look right,

And yet it thinks,

And yet

Did you switch from having one that was managed by, oh, it should be deleted then. No, I was gonna say, um, did you move from a four each or to a four each from not a four each or stuff like that? Is that potentially. The problem, I don't, no. I've had that before where it's tried to like, create something before it destroyed the old version and they happened to have the same name, and it didn't realize that they were gonna stomp over each other. But that doesn't sound like this

No. Dvo records source name, value and type. Uh, you know, the other thing I'm gonna do is I'm gonna open this up in, uh, some Jet Brains tools so that I can get the Terraform plugin to tell me if I've done anything consciously stupid.

But the thing is, Terraform would tell you itself, right. You know, Terraform validate and Terraform itself plan will do at least its side. Usually the problems come when it tries, when the rubber hits the road and it doesn't know it, it doesn't properly model what the provider is going to do when it actually applies these things. Mm-hmm.

All right. Here's what I'm gonna do. Yep. I'm gonna comment out the verification. No, I, the two ones that we need are the, the, the top level domain and the dub dub dub. So I'm gonna comment out the record for the verification, cuz we did that manually once already. Yeah.

And we can always just blast those all the way in both the console and here or whatever. Yeah. Let's apply this. Let's try and get the closure of knowing that it applies cleanly and then I think we're pretty much done here.

Yeah, yeah, yeah. Yeah.

Did that apply cleanly now? Is that, will that you It's gone. Ah it's going.

I mean, we'll see. We'll see. But it's, it's trying to, it's trying to do it 20 seconds lapse, stay on Target,

.

Oh, console's looking good though. I got two records both point to the cloudformation

That's what we wanted to see. I mean, I'm seeing that on my side as well here. So I think we're, we're

There. Yep. Yep. And yes, Terraform apply complete.

Complete. Okay. Then I think Awesome. Think we can declare almost complete victory at this point. We don't fully understand why those, those other records were either they're not there or whatever. Maybe Amazon's putting them in automatically as well as you trying to put them in manually or something like that. That would be my guess now because it's, it's managed by them already. Um, so you just, just leave them out and then Terraform never needs to know they exist. Right. And it'll just work. So final work for this then is to, uh, double check the certificate Looks good. Which I think it probably must do. And then point the top level domain registrar at aws. Change the domain, the DNS records, um DNS servers. Sorry to put be Amazon's ones and or move the whole thing. It's up to you how you own them.

Yep. No, and then that should,

And then finally we should bump the TTL back up to something kind to, uh, everybody. That's the other last thing that No, everyone forgets myself included, is that like, well if you don't need it to be 60 seconds, then it might as, I mean who knows who say anyone pays attention to these TTLs properly anyway.

. Right, right. Cool. Cool,

Cool. Well, there we go. We got success. We did it.

We got success.

Hopefully by the time this airs, people will actually be able to go to https to compliment org and it will just work.

It'll just work. Fabulous. Awesome.

Okay, my friend. Until next time.

Until next time.