Welcome, everyone, to Kaizen 6. It's slightly different this time, and it's for you to decide whether it's better or worse in this format.
Jerod Santo:Or how much better.
Gerhard Lazu:Or how much better, exactly. Just me and Jerod this time; we're keeping it cozy... Even cozier than last time, Lars...
Jerod Santo:Ooh...
Gerhard Lazu:So Lars told us, feedback, in our Slack channel, that episode 50 was a very cozy Kaizen. Nice conversation, collapsing into laughter, and lots of assorted TikToks. So I'm thinking more laughter this time, because we don't have serious Adam, so we can go crazy... There's no one to stop us.
Jerod Santo:Right. We can go crazy. He'll never try to bring us back to center.
Gerhard Lazu:Yeah, exactly. If we go off the rails, we will not recover from that. \[laughter\] So yeah, cozier. We're getting cozier, Lars. And we blame you for getting cozier, because you gave us the idea.
Jerod Santo:What does cozy mean exactly? Because I'm thinking like fireplace, bearskin rug, coffee...
Gerhard Lazu:I didn't ask him, but I think he may have been referring to that Swordfish scene... \[laughter\] I don't know. Lars, can you clarify for us, please, in the comments below, what do you mean by cozy?
Jerod Santo:Yes. Just how cozy is this getting?
Gerhard Lazu:But I think just like getting closer. Like, closer.
Jerod Santo:Okay. Intimate.
Gerhard Lazu:Not as close as Swordfish. Yeah, exactly.
Jerod Santo:\[03:59\] But not that intimate.
Gerhard Lazu:Not too intimate.
Jerod Santo:Okay.
Gerhard Lazu:So I want to start -- I could hardly wait; it's been a week since we had this, but I've been mentioning to everyone...
Jerod Santo:You're pretty excited about this.
Gerhard Lazu:...my favorite biggest improvement... I'm really, really excited about this. So we have our own vanity domain.
Jerod Santo:And that domain is...?
Gerhard Lazu:Shipit.show.
Jerod Santo:Shipit.show.
Gerhard Lazu:I'll repeat that, Shipit.show.
Jerod Santo:Pretty cool, right?
Gerhard Lazu:So someone, especially non-English speakers, they thought I said s\*\*t show. I didn't say that. \[laughter\]
Jerod Santo:We did not say that. This was your concern from day one, wasn't it?
Gerhard Lazu:Shipit.show it is.
Jerod Santo:Yeah. Have we had any complaints about that particular aspect of the show so far? This probably crossed nobody's mind until just now... And then we're like "Oh, man...!"
Gerhard Lazu:Oh, dang it!
Jerod Santo:Now they're gonna hear that every time you say it.
Gerhard Lazu:Okay, we may need to cut that out. \[laughter\] So Shipit.show, and if you do Shipit.show/60, that will be this episode.
Jerod Santo:Boom.
Gerhard Lazu:How nice is that?
Jerod Santo:So cool. Let's talk about vanity domains writ-large for a minute, because it's like... Short URLs - they used to be cool. Remember tinyurl.com? Going way back, there was a time when URL size mattered... And shorter was better.
Gerhard Lazu:Yes.
Jerod Santo:And for us it's still true, but for most people, nobody cares anymore about that, right? I think it was probably because of Twitter... It used to be that your long URL inside of Twitter counted each character in the URL; it counted toward your character count. And we had 140 back then? What did we have? Yeah, we had 140 characters.
Gerhard Lazu:Yeah. Then they went to 160, but you're right, links were not expanding; they weren't shortened, so that wasn't great.
Jerod Santo:Yeah. So if you had a really long URL and you were trying to link to it, you couldn't even put any content into the tweet. So tinyurl.com I think was the first one that I ever used... And then bit.ly became huge. And bit.ly had a really cool feature where you could do, of course, bit.ly/whatever. But you could also sign up and get your own custom domain. And that's where us nerds really started to nerd out, in like "What's the shortest, coolest --" I tried so hard to get san.to... For Tonga, because there was the .to domains. And somebody in Tonga owns it, and I have a recurring reminder to email them once a year and be like "Hey, can I have it now?" Because this still would be cool, right? San.to.
Gerhard Lazu:That's a cool one.
Jerod Santo:But nobody cares as much anymore, because now Twitter has it built-in, pretty much every social media thing is building it in... And it just has not been all that necessary. But when you podcast, when you try to point people to places audibly, and you have to say "Go to changelog.com/ship-it/60", it's so much cooler to say "Shipit.show/60".
Gerhard Lazu:Yeah, that's true.
Jerod Santo:Now, we held off on this one for a while, because we were trying to get shipit.fm... Which actually - you like this one better, don't you?
Gerhard Lazu:I do, because I like to think of this as a show. I really do.
Jerod Santo:Right. Yeah, for us it was just consistency, because we've used fm for everything else. So every one of our other podcasts had fm... And we've spent -- I mean, it's probably been a year now trying to get it. And then also trying to get ship.it, which we couldn't get either... But Shipit.show is cool.
Gerhard Lazu:I think so, too. I think it's a good third option, which was also available, and that's what matters in this case. I mean, we're still open to getting shipit.fm. So if someone knows the person that has it, or if you are the person that has it, and you have a reasonable ask price-wise...
Jerod Santo:Yeah, we've got money... Though not that much money.
Gerhard Lazu:\[07:53\] Exactly, yeah. We're not millionaires, yet. When we will be, we can pay you more if you want. Or even better, if Ship It makes many millions, we don't mind. But for now, let's just be reasonable. Shipit.fm would be cool. Ship.it would also be cool, but it was a bit more complicated. Actually, we didn't get any replies.
Jerod Santo:Super-short...
Gerhard Lazu:Yeah. We tried, we didn't get any replies. And now Shipit.show - for me, the most important thing was when you announce those new Ship It episodes in the Slack, we don't unfurl the links, which means that it's just a link and you don't know what it is... But if you look at GoTime.fm, or JSParty.fm, or even Changelog.fm, you see a preview of the episode. And I really miss those, because it's so easy to miss when a new episode gets announced.
Jerod Santo:They are nice. And the reason why it doesn't work without the vanity domain is because if we'd unfurl every changelog.com URL, it'd be like the same thing over and over again. So we just permanently banned unfurls on the changelog.com domain. But it's nice for the podcast episodes, because they actually have a pretty nice unfurl, which shows you the album art, it shows you the title, the description... But those are unbanned on vanity domains, and Ship It, for the first 57-58 episodes (I can't remember when we turned it on) did not have that. And Gerhard - it was just eating you alive, man. It was just eating you alive.
Gerhard Lazu:It was, yeah. I really wanted that. Like "Seriously, surely this Kaizen...? Nope. Not this Kaizen. It's not Christmas yet."
Jerod Santo:"No, not this Kaizen..." So maybe we talk very briefly about how these things work, because it sounds like there's a problem... I think I know what the problem is. You put in our notes there's a "Something went wrong" page if you go to shipit.show/61, for example, once this one goes live... It's because there's no content there.
Gerhard Lazu:Yeah, the same thing happens for every episode.
Jerod Santo:Every 404, I think, actually.
Gerhard Lazu:And maybe every 404, you're right. Yeah.
Jerod Santo:So I was digging into that a little bit... It's like we're serving the 404 header response. But for some reason we're not serving the content correctly, the actual template... So that's in the app. I thought it maybe was like a way that Fastly was not doing it right, but I took Fastly out of the equation and it still happens. So I think there's just something -- it used to work. Somewhere along the line, the way we serve the 404 HTML just doesn't... Like, it tries to download as a -- maybe the mine type is wrong. I don't know, I have to look into it. But that's really what it is. It's not the vanity domain that's a problem, it's not Fly, it's not Fastly. It's just like the app serves the 404, but can't serve the content for a 404. So it just tells you something went wrong.
Gerhard Lazu:I always thought that was an improvement that we were waiting someone from the community to do. It wasn't really an Easter Egg. We knew about this for a while; at least I knew for at least one year. I've seen it before. But I always thought that someone will pick up on it and will want to improve it. It's a great small improvement. It hasn't happened... So at what point do we improve it ourselves?
Jerod Santo:Yeah, exactly. Or put a bug bounty out there, or something.
Gerhard Lazu:Yeah. I mean, November is still far away, but that would be a nice one for November.
Jerod Santo:Oh, you mean for Hacktoberfest?
Gerhard Lazu:Yeah.
Jerod Santo:That's October though... \[laughs\]
Gerhard Lazu:Oh, October. Sorry. I'm thinking Movember... No. Mustache? No, no, no.
Jerod Santo:No mustache. \[laughter\] Yeah, no bounties for mustaches. But yeah, maybe we could put something out there. I mean, honestly, it's probably a 15-minute fix once I actually go into it... I just haven't -- I've known about it for a while as well... I just can't be bothered sometimes. I'm like "So it's a 404... You're not a page that exists..." I would much rather have a nice thing there, but... Now that it's on Kaizen, I'll probably fix it, darn it.
Gerhard Lazu:Yeah, Kaizen-driven development. KDE.
Jerod Santo:Yeah, Kaizen 70... Dang it. Added to the list.
Gerhard Lazu:I really like that. \[laughter\] That's cool.
Jerod Santo:That's how I do most of my development.
Gerhard Lazu:Yeah, Kaizen-driven. I love that. Me too. I try to, as much as I can. So Shipit.show/50 does work, because it's already out... That was the previous Kaizen.
Jerod Santo:There's a lot of cool -- I pasted you a link... Maybe we can link into the codebase where it does this as well. The way that vanity redirector works - there's a bunch of other cool little URLs you can use.
Gerhard Lazu:Ah, yes.
Jerod Santo:\[12:00\] So like Shipit.show/apple will get you to the Apple podcast URL, which is much longer. So this is like a classic shortener. /spotify. /android. /merch if you wanna get to our merch shop. These are just nice ways to just have it in your mind when you're like talking to a friend and you're like "How do you listen? On Apple?" "Yeah." "Shipit.show/apple" will get you to the right spot, and you don't have to know that. So we have like 10-15 of those, which are pretty cool, I think. I use them quite often. And we link to them on YouTube, and in our Twitter pinned tweets, and stuff. Just clean.
Gerhard Lazu:One URL which reminded me of something that one of our listeners gave feedback on is Shipit.show/request, which people can use to request an episode, or propose an episode. It works the same way. So this was - and I'm trying to find it - \[12:51\] He said "Why on Earth would I possibly want to create a Changelog.com account just to send you guys a little tip, pointer, suggestion for an episode? There's simply no point in not letting people come up with suggestions easily, without having to go through hoops for it." So what do we think about that?
Jerod Santo:Well, I have two responses. The first one to Leo is "How did you submit this feedback?"
Gerhard Lazu:An email.
Jerod Santo:So just write your suggestion into that email. I'm being spicy a little bit, but my point is, there's ways to reach us. You can tweet at us, you can email us, you can be in our Slack... We are very open to communication. This is an official channel, and we get tons and tons of these requests. I mean, hundreds. So while it's a little bit of a pain on your end to sign up and do the submission form, that's kind of on purpose... To just put a little bit of a barrier between us and requests, just because we get so many.
Gerhard Lazu:How bad do you want it, Leo? If you want it badly, you will create an account. Okay, so again, still spicy; I'm continuing down the same alley. But let's try a different one. From my perspective as a show host, I really like when I see those requests in the admin area, because it allows me to -- when I basically create a new episode, I can use the request that you created to generate the episode. And that helps me. The other thing that helps me, and I think it basically helps you, the ones that request the episode, is you'll get notified when the episode goes out. I'm not sure whether we notify when the status changes for a request...
Jerod Santo:Not automatically. So you can decline a request and send a message. There's multiple ways. You can decline it silently, you can decline it with a message and let them know "This didn't work out. Here's why." You can also fail it with a message, and that's like handwritten. So it's basically a markdown-based email that we send out, that says "This one didn't work out. We tried to make it." I just sent one of those today, actually. We had a request from two years ago. I was working on it. I had a yes, we had a reschedule etc. life came up, pandemic hit... It fell by the wayside, and I finally failed it today with a message that was like "Look, we worked really hard on this, I wanted to do this show; it's just not working out. Maybe another time." And we can send that message back. But when you set it to accepted, or you set it to "scheduled" and stuff, we don't wanna send people a bunch of emails, plus it still might fall apart... So the only thing we send is if you tell them "I wanna send a message" or when the show does go live, we automatically send an email at that point.
Gerhard Lazu:So in summary, Leo, if you want a quick suggestion, you can do it via Twitter, @ShipItFM, @Changelog, @gerhardlazu... All work equally well. You can also send an email, as you have, which is gerhard@changelog.com. I always read those. Or any other email that is out there from me. It will still get to me. And if you go via the website, and if you do create an account, it helps us to keep track of those, it helps us to basically manage them better, and then you will get a notification when the episode goes live, when the episode goes out as a request episode. I like to see those nice "requested" tags on them. I find that helpful. And the last one which we had - and it just shows how well I remember it - was the Docker Swarm request. I forget who exactly requested it, I forgot the name, but it's very easy to go and see that... But that's the last one that was requested, and it's just a nice flow.
Jerod Santo:\[16:23\] Yeah, it's really nice. And sometimes people will email in, like for the Changelog specifically, and have an idea... And they'll actually say "This is a good idea. Can you go open up a request on the website?" Because that way it actually sits in our queue of ideas, right there in the admin, with everything else. It just has a much higher likelihood of becoming real, versus on Twitter or versus an email. But those are still avenues for starting a conversation. So Leo, you don't have to create an account. You can try those other ways. But that's there for reasons, but for our workflow, and also just as a little bit of a bump in the road for people who would spam us otherwise. If it was just completely open, we'd get hundreds a day, versus - now we're getting hundreds a month.
Gerhard Lazu:Cool. Now that we addressed that - and I'm happy how we addressed it; thank you, Jerod - we can go back to talking about episode 50, and what other topics did we cover since episode 50.
Jerod Santo:Okay.
Gerhard Lazu:For me, the big ones were episode 51, Shipit.show/51 (you already know how this works), with Mark Ericksen, where we talked about the clustering part and the multi-region PostgreSQL integration, which I was convinced I will do by episode 60, and it hasn't happened. It just shows reality versus plan. And this is normal.
Jerod Santo:Yeah.
Gerhard Lazu:With other things, which are more important - we'll get to them in a minute. But that's the one thing that we talked about, I didn't have time to action on. Anything that you wanna mention about episode 51, Jerod? First of all, did you listen to it?
Jerod Santo:I did.
Gerhard Lazu:Okay.
Jerod Santo:I listen to most of them. I haven't listened to 59 yet, which you're gonna talk about next... I listened to 51, I enjoyed it quite a bit. I think the biggest surprise to me during that one is that Postgres is not really managed, like I thought it was. It seems -- I wouldn't wanna call it an afterthought, but it's kind of like... It was a good idea, of like "Well, we have these runtimes, and it's just this version of one." It's just like anything else - you have an app container... What do they call them? Containers? Runtimes? Pods?
Gerhard Lazu:I think it's like an application.
Jerod Santo:Okay, apps.
Gerhard Lazu:I mean, the application in like the Fly context - there's a special one, PostgreSQL, where you can create a cluster, and the Fly CTL, the Fly CLI, it's slightly more integrated, you can scale them better, you have visibility into back-ups... Things like that.
Jerod Santo:Yeah, it's kind of like any other Fly app, but with special privileges. Special benefits.
Gerhard Lazu:Correct. That's right.
Jerod Santo:And we were thinking more as like "This is our Postgres service that we're gonna manage for you." And there is some management going on, but it was less formal, and maybe a little bit less than I was expecting, and kind of hoping for.
Gerhard Lazu:Same here.
Jerod Santo:Yeah. So that's what I remember from that episode. And then I remember you giving an ode to Erlang at the end, which I very much enjoyed.
Gerhard Lazu:Yeah, I remember that coming out like as a special -- I've seen it.
Jerod Santo:Yeah. I put that out as a clip.
Gerhard Lazu:That's how I know that you listen to parts of it, because there's a clip appearing on Twitter, and I know that that's something that resonated with Jerod, and he thought it was good enough to share. That's how that works. So related - again, this is the episode which hasn't come out yet as we record this. That is episode 59, with Ben Johnson.
Jerod Santo:Right. Very much looking forward to that, by the way.
Gerhard Lazu:Oh, that was a good one, yes. SQLite instead of PostgreSQL. That was a great one.
Jerod Santo:I guess by the time people are listening to this, they've heard it. But I haven't heard it. So you can't really tease it for them in like past tense, and I haven't heard it yet... So I don't know, just move on? \[laughs\]
Gerhard Lazu:That's exactly what I'm thinking, move on.
Jerod Santo:Okay, moving on, moving on... Fair enough.
Gerhard Lazu:Generating some interest for you... You know, trying to prepare you for it when it comes out, because I thought it was very good... But it's more like future-looking. Again, I don't want to spoil the fun for everyone, but it's more future-looking for us, and it's also some important things that we would require in Fly, in the context of Fly, for us to be able to use it... Which is coming, but it's not there yet. So that's great, we know that we're about to ride the wave which hasn't even appeared. My favorite thing.
Jerod Santo:Yes.
Gerhard Lazu:So yeah, we can move on.
Break:\[20:26\]
Gerhard Lazu:Speaking of moving on, one thing that we definitely moved from is our LKE, our managed Kubernetes. Because 30 days after the migration I deleted -- I won't say everything; I definitely deleted the deployment, and everything that we were running on it... And one thing which I wanna mention is the importance of keeping things around just in case. So the reason why we haven't deleted it just then, or shortly after, is if there's a problem and we need to go back, I wanted to make sure that we have something to go back to. So I'm thinking of it as a very long blue/green. Think months. In our case I think it was like two months. So if within two months we realize that this plan is not working as we think it is, reality vs plan, we always have the option of going back, or stopping the plan, depending on where we are. So a month before, as we were standing up Fly, we were still running on the old one, still running on LKE, which was the current production, and after we went to Fly, we always had the option of going back. And that was like a very deliberate choice. Luckily, we didn't have to use it, but it's one way to make sure that you always have something to fall back on. And I keep mentioning my love for running two things - at least two things - and this is what it looks like in practice.
Jerod Santo:So how did you pick 30 days or 60 days? Because for me, always the question is "How long do I end up running this?" And then I turn into a hoarder, and it's like six years later and I've got all these services that I haven't been using for a long time, but I'm still running them just in case.
Gerhard Lazu:For me it's the Kaizens.
Jerod Santo:Okay...
Gerhard Lazu:So that's why -- the recurrent theme is really powerful in a lot of things that I do. And the reason why I have these Kaizens is because they force us to do certain things.
Jerod Santo:So what do you say to somebody who doesn't have a cadence? Like, they're not gonna record a podcast once every ten weeks to Kaizen.
Gerhard Lazu:True.
Jerod Santo:How do they do it? They can't count on --
Gerhard Lazu:Get a podcast and record it every ten weeks. \[laughs\]
Jerod Santo:Yeah, everybody needs a podcast. "Get a podcast" is the solution.
Gerhard Lazu:Yeah, that's the conclusion. \[laughs\] So to be honest, I would set myself a reminder, whatever that looks like for you. So remind yourself to do something in the future. Whatever app you use, whatever system you use, set yourself a reminder to do something.
Jerod Santo:Even if that reminder is simply to reevaluate. It doesn't mean you're necessarily going to delete it, right?
Gerhard Lazu:Exactly.
Jerod Santo:You're like "Okay, after 30 days I'm gonna have this thought." Maybe it takes 30 seconds, maybe it takes half an hour to actually go ahead and execute on it... But remind yourself to think about it later. "Is it time to delete this now, or do we wanna keep it there for another 30?" It's usually a pretty easy decision to make at the time. What's hard to make is like looking forward six months from now etc.
Gerhard Lazu:Exactly. And I think people kind of know when it's been long enough. So for some, it may be seven days, for others it may be a month. Others still may need more, a longer period. And that's okay. All those options are valid. In our case, it is arbitrary, I have to say. 30 days - it just so happened, it was roughly 30 days. Maybe it would have been 31, 32. I can't remember exactly the time.
Jerod Santo:Sure.
Gerhard Lazu:But there is transition period which starts whenever the new idea starts, and you still have to run your old system or your current system until you're migrating on to the new one. But when you have migrated, your work isn't finished. The migration is done, in that you're running on the new system, but the old system is still around, because you want to give yourself a plan B if things go south. And it has happened for us. I'm not going to go into details, but episode 50 - we were delayed by a week, or two weeks, something like that. So in my case, I kept it around just in case we may need to go back, just in case we discover something that we didn't know, until we switched across. And for me, after 30 days it was just a reminder, because I was thinking about the Kaizen, and what to do next... And this is actually linked to our TSL certs. So we had cert-manager running in LKE, and cert-manager was syncing -- actually, first of all renewing the certificate for the wildcard one for changelog.com. But also we had -- I wanna say the job... Was it the job? Yes, it was the job which was keeping it in sync with Fastly. So I knew that I had to migrate the certificate somewhere, and what I did - I just delegated the management to Fastly for the wildcard Changelog.com certificate. There's a limitation in that only one provider can manage the wildcard certificate, because in DNS you end up creating CNAME records, and it can't have multiple; it can only have one. So there's that limit. We'd have happy with text records but there you go it's just a limitation of how that's implemented. So we cannot use cert-manager or Certbot and Fastly. We could only have one. In our case we had cert-manager, because we were managing all our certificates in LKE, using cert-manager, for all the vanity domains.
Jerod Santo:Right.
Gerhard Lazu:\[26:25\] But when we migrated to Fly, the easiest thing was to let Fly manage the certificates for the vanity domains. So shipit.show, for example, to configure it was super-easy. Really, really easy. And the wildcard ones - I can't remember why we don't do it. Oh, yes --
Jerod Santo:Because you have to upload it, right?
Gerhard Lazu:Exactly, because something needs to upload it. So then we have to manage that.
Jerod Santo:Fastly has to have it, and Fly has to have it, if it was going to be managed by Fly.
Gerhard Lazu:Exactly.
Jerod Santo:Whereas if Fastly just manages it, Fly doesn't care about it, for the application to run.
Gerhard Lazu:Exactly.
Jerod Santo:So we have *.changelog.com is on Fastly, and everything else we do, which is mostly the vanity domains, is on Fly, in terms of DNS -- or not in terms of DNS; in terms of certs.
Gerhard Lazu:Yeah, correct.
Jerod Santo:That's not too bad. I mean, it's not super-clean, but it's not super-dirty either.
Gerhard Lazu:The thing which we're missing more is documenting that... Like, to basically capture that. And this goes back to episode 44, with Kelsey, where he mentioned that.
Jerod Santo:Kelsey would be so upset with you on the documentation front.
Gerhard Lazu:I know. I'm sorry, Kelsey, I'm still working on that. I want to show you my list; I know that you wouldn't care, but it's there, trust me. \[laughs\]
Jerod Santo:Here's a workaround. All these Kaizen episodes - you know we transcribe the entire conversations. So in a way, you've just documented it right now.
Gerhard Lazu:True. That's true. But it's spread across like four or five different episodes.
Jerod Santo:Well, he didn't say the documentation needed to be good, he just said you needed to document it, right?
Gerhard Lazu:That's true, that's true...
Jerod Santo:So we're working on it...
Gerhard Lazu:Yeah, you're right.
Jerod Santo:Just speak it out loud and it documents itself. It's self-documenting podcasts.
Gerhard Lazu:I like that. So I just need to record myself waffling for like two hours, and then you have all of it. \[laughs\]
Jerod Santo:So once again, the solution is everybody needs a podcast. Their documentation needs would be taken care of.
Gerhard Lazu:There you go. We just keep coming back... I mean, only if we didn't own a podcast, and a podcast network...
Jerod Santo:Right.
Gerhard Lazu:And you need to run that podcast on our infrast4ucture. That's the other requirement.
Jerod Santo:That's right. So you talked about having two things hanging around, and how long it's gonna stay that way... And I do tend to be the cut it off too early type more so, and I have regretted that in the past... But I'm somewhat of a purger. I like to purge. I don't like things that infinitely grow, such as blockchains, and Twitter.com... \[laughter\] Because the tweets just keep coming; you're never gonna get to the end.
Gerhard Lazu:What about podcast episodes? Do you like those to keep growing? \[laughter\]
Jerod Santo:No, they need to stop at some point.
Gerhard Lazu:When do we reset the Ship It number?
Jerod Santo:Right...
Gerhard Lazu:With some exceptions. This is one.
Jerod Santo:Yes, yes. Podcasts should just always keep going. You have two directories - 2021, 2022, and there's like a 2022.fly... And then there's others, Junk, going on... \[laughter\] And I just wonder --
Gerhard Lazu:Don't call it junk
Jerod Santo:Okay, one man's treasure is another man's trash... Or I can't remember how that saying goes. But for me, the only reason why this bugs me is because I need to copy the fly.toml into the root directory, unless I wanna have another tmux session in the subdirectory... And I don't. I don't want another tmux session. So I'm wondering why we need to have those years anymore. I feel like we're kind of beyond that. I'm just curious when is that whole thing gonna get cleaned up.
Gerhard Lazu:I see. Okay. \[laughter\] That's a great question, Jerod. I'm so glad that you joined this episode...
Jerod Santo:Very cozy in here. It's getting very cozy.
Gerhard Lazu:Right. Yeah, because we're talking about junk, so you have to, right? \[laughter\] There's no way to talk about it without getting cozy.
Jerod Santo:Oh, my...
Gerhard Lazu:Right. So...
Jerod Santo:We need to put that Explicit tag back on this one...
Gerhard Lazu:Where is that? I think we need it now. \[laughter\] I think Adam is talking out loud, and he doesn't realize he's not part of the podcast as he's listening to this...
Jerod Santo:It's getting hot in here.
Gerhard Lazu:\[30:04\] So 2021 - there's one thing that we still need there... So while I did delete the infrastructure, the directory is still there, how we did the config. I just found myself referencing something for James Haar. So James Haar was following up on episode 58, the one where we talk about how to keep a secret with Rosemary and Rob, the Vault episode... And I gave him an example in the 2021 directory of how we integrate LastPass, secrets in LastPass, with Kubernetes. Or how we used to do that. So it was helpful for that reference.
Jerod Santo:Gotcha.
Gerhard Lazu:But I could have done it in a number of ways. I didn't have to keep it around. So in the 2021 directory, the most important thing is actually our CI integration. So we're running Dagger; that runs locally. It also runs GitHub Actions. And if you remember, we did that to migrate from Circle CI, and that was episode 33. So that's when that happened. So the only config that we have is the Dagger config, which describes everything that needs to happen in our CI. That config is pinned to Dagger version 0.1.0. Since, Dagger 0.2.0 came out, and that changes a couple of things. I didn't have time to rewrite the config from 0.1 to 0.2. I should have, but there were other things which kept trumping it. So that's the most important thing that would need to move from 2021, from that directory. 2022 was for the new Kubernetes cluster, which as you remember, that's the one that we had around when we did the migration.
Jerod Santo:Right.
Gerhard Lazu:We felt we'd go to Fly, we couldn't go to Fly, episode 50 has all the details... We ended up on this new 2022 Kubernetes cluster. So we could delete that directory, we don't need it anymore, but what we do need is the 2022 Fly. That directory contains all the config for the Fly setup. The reason why I use the year - because if you remember, every year we used to upgrade those.
Jerod Santo:Yes. Legacy. It's legacy.
Gerhard Lazu:Exactly.
Jerod Santo:Fair enough.
Gerhard Lazu:And then we eventually would use to delete them. So I think that we will be able to delete the 2021 as soon as I migrate Dagger. By the way, 2022 Fly is already using the new Dagger, and I tried to configure it to use with Fly. But as I was doing the migration to Fly, I realized that there's quite a few things which I need to figure out, and I had to separate the work that I had to do on Dagger, versus the work that I had to do for this migration. So at some point I said "Okay, what's more important?" So I refocused all my energy and effort on the migration, and I left the Dagger integration with Fly secondary. So the Dagger in the 2022 Fly is already at 0.2. That means that the two are incompatible. So until that migration happens, I can't change it. Another thing which happened in the 2022 Fly directory - I've added the Docker Engine integration. This is a nice segue into pull request 416, where I explain why we had to deploy and use Docker Engine on Fly.io.
Jerod Santo:So this is the one where we had those connecting to a Tailscale that you were running at your house, or in some --
Gerhard Lazu:Exactly. It's right there.
Jerod Santo:Yes.
Gerhard Lazu:I'll put a picture. I have a fanless NixOS bare metal server. There's no fan; not even in the PSU. I've been waiting to do an episode on that.
Jerod Santo:No fans.
Gerhard Lazu:Yeah, no fans.
Jerod Santo:Is there a heat sink on the CPU?
Gerhard Lazu:Yes.
Jerod Santo:\[laughs\] I thought maybe you were going super -- no cooling.
Gerhard Lazu:Not 2S--though, that's actually what makes me hot in this room. \[laughter\]
Jerod Santo:Here I thought it was all of our cozy talk...
Gerhard Lazu:\[33:56\] Yeah. \[laughs\] It's that fanless server. So anyways... It was connecting via Tailscale to that host. The problem with Tailscale is that when you generate an auth key for the GitHub Actions runners, the maximum TTL is 90 days.
Jerod Santo:Oh yeah, this one bit me. I was trying to deploy some stuff --
Gerhard Lazu:A couple of times. Gone.
Jerod Santo:Yeah.
Gerhard Lazu:Tell us about it.
Jerod Santo:I had to go "Gerhard...!" Well, I'll tell you what happened - it wouldn't deploy, because Tailscale couldn't log in, or something... Or couldn't connect, I don't know. It would fail on the Tailscale, and you would, you would see either TLS, or SSL, or who knows; some sort of error right there. And I said "Gerhard...! I know this is running in your house! Help me!"
Gerhard Lazu:Yeah, "Fix it!" \[laughter\] Exactly.
Jerod Santo:"I'm coming over!"
Gerhard Lazu:Yeah. So that was that thing... If I knew that you were coming over, I wouldn't have fixed it, to force you to come over. \[laughter\] Not that you would have known how to fix it, by the way, because there's two other issues...
Jerod Santo:Oh, okay. Yeah.
Gerhard Lazu:So let me tell you about the other two issues.
Jerod Santo:Oh, okay... There's two others...
Gerhard Lazu:So there is a Docker Engine running on that host; it's a bare metal host, so very beefy, very fast... It has an amazing SSD, an NVMe 1, NVMe Samsung 980. Super, super-fast. 64 gigs of DDR4, and a Ryzen 3, 16 cores, I think. Or 12 cores. I can't remember. One or the other. Anyways. So it's really fast. The reason why we do that is because right now - and this is something which we're fixing in Dagger - the caching doesn't work for volumes. Out of the box it doesn't work. What that means is that in our application we compile a lot of dependencies, in our Phoenix application. So to compile those dependencies, if you don't have a cache so that you can mount all the compiled ones, it would take five minutes easy. Maybe even more, because we know that the GitHub runners only have two CPUs by default, so they're really not that beefy. So we want a Docker Engine that is persistent across runs. When you start mounting volumes in GitHub Actions, if you've done it, you'll notice there's all sorts of issues now and then. So your CI isn't as stable and reliable, because GitHub Actions are meant to be the runners, they're meant to be ephemeral. And if you have state, to recreate that state, to get it back from the cache, to restore it, it can be slow. Sometimes things fail, because it's a distributed system, and sometimes it fails... And then your CI becomes less reliable. So the best thing to do is to have somewhere a runner that you can trust, which is not ephemeral. It will only work for your CI, and in this case it's not the runner, it's actually the Docker Engine itself. But this Docker Engine is running in Fly, it's a Fly application. Thank you, Kurt, for the starter. I even mention it in pull request 416... So in our 2021 Fly directory we have the config that we used to deploy the Docker Engine on Fly. So now, whenever GitHub Actions runs using Dagger, it connects to this Docker Engine running on Fly.
Jerod Santo:Okay.
Gerhard Lazu:So you get the caching, you get the volume, you get all that stuff; it's just as fast. The difference is that it's using WireGuard to connect to your Fly app. And by the way, if you're looking at the WireGuard GitHub Action in the marketplace, that one didn't work for me. So I had to basically follow the Fly.io instructions on how to connect the GitHub Actions runner to the Docker Engine which is running on Fly, and that worked. And we are using that for our app, so you can see that.
Jerod Santo:I can see that. Let's say we have a dedicated CPU 4X, with 50 gigabytes of storage running there.
Gerhard Lazu:Yeah, exactly. And that's for the caching, and everything. And there's something really cool. There's this really cool -- again, we are mentioning all these things because I'm excited about them, and I haven't tried it out yet, but I can hardly wait to try it out. \[38:00\] So Fly introduced Machines, which can spin up in milliseconds. So imagine VMs that start really, really fast. So if you think about our Docker Engine, the one that's there for the CI, we're using it less than 1% of the time. So 99% of the time it's running, it's using those --
Jerod Santo:For no reason.
Gerhard Lazu:...for no reason, exactly.
Jerod Santo:Should we set up a SETI@home, or...?
Gerhard Lazu:We could do. Or a Bitcoin miner. \[laughter\] We could. It's there, we're paying for it... But you know, to be honest, the reason why this is exciting is because we can spin up on-demand, and we can have maybe more than four CPUs. We only chose four CPUs because of the cost. It just hits the balance nice. But if we could use 16 CPUs on-demand, that would be a lot better. And with Fly machines we can, because they spin up so quickly... You don't wait minutes for it to come up.
Jerod Santo:Like, really? Really that quickly? What about the disk? Is it like connected? You just connect it to some sort of --
Gerhard Lazu:There's a blog article that says 250 milliseconds. If it takes more than that, I think we should complain to Kurt. That's what I think we should do. \[laughs\]
Jerod Santo:I think so, too. Okay...
Gerhard Lazu:Again, I haven't tried it out, but I really do want to. I think that's like an improvement to make on this. The first improvement was to move it off my server, because as I mentioned, the Tailscale key would expire, I would need to manually renew it...
Jerod Santo:Yeah.
Gerhard Lazu:There was the other issue - because I'm using it for my development, it's Linux-based, so it's NixOS... So when I'm running my development version of the Dagger CLI, that may need a new --
Jerod Santo:Wait, are you telling me this is your dev box?
Gerhard Lazu:Yes, it doubles up as my dev box
Jerod Santo:Your dev box was part of our critical infrastructure for months... \[laughs\]
Gerhard Lazu:Well, critical... I mean, if it doesn't work for me, it's more important. It's something that -- I basically use it often. So my dev box has actually three hosts, okay?
Jerod Santo:\[laughs\]
Gerhard Lazu:I have an iMac Pro... So it depends on the day.
Jerod Santo:So you moved on from "Have two of everything." You wanna have three of these.
Gerhard Lazu:Three of these, exactly.
Jerod Santo:Yeah, three is better than two.
Gerhard Lazu:I have a MacBook Pro, an iMax and the NixOS. Exactly, yeah. \[laughs\] Yeah, and it's running a bunch of other things. The point being that sometimes when I would develop -- so I don't run Docker on my Mac.
Jerod Santo:Too slow.
Gerhard Lazu:Yeah. That's basically it. The TL;DR is "Too slow."
Jerod Santo:That's why I didn't do it, yeah. Too slow.
Gerhard Lazu:Yeah. And even now, it's still too slow. Especially in networking... Anyways, really weird, because of how it works with the virtualization on MacOS. So Docker and Linux - all the way, but not anywhere else. I didn't even try it on Windows, but I imagine it's just as bad, if not worse. Anyways... So because Dagger manages its own BuildKit, I was constantly upgrading the BuildKit that Dagger 0.1 was using to the latest version as I was developing Dagger... And whenever the CI would run, it would downgrade it, and I would upgrade it, and it was like this constant battle of "Well, which version should you run?"
Jerod Santo:\[laughs\] Oh, gosh...
Gerhard Lazu:Okay, you can configure it, you can specify a new thing... I haven't. That was the other thing which bit me. But again, the fix was so quick that by the time you realize it was an issue, I already fixed it. And because I was using it -- so again, it's an issue that was happening, but you weren't aware of it, because I would always fix it.
Jerod Santo:Now, are we still vendoring Dagger in our source code?
Gerhard Lazu:We're not vendoring Dagger as the packages...
Jerod Santo:The Dagger packages. Is that common practice, or is that because we're on edge, or bleeding?
Gerhard Lazu:We haven't basically figured this problem out in Dagger, but right now any Dagger package that you use, you're supposed to basically link them to the Dagger CLI version, and you do that by running dagger project update. And then anything that you're running locally -- so it doesn't have package management like Go, or Python, or some other language.
Jerod Santo:Isn't this built in Go?
Gerhard Lazu:Dagger is built in Go, yes.
Jerod Santo:Can't you just universal-binary that thing? Or I guess there's too many packages. You wouldn't wanna throw them all in there.
Gerhard Lazu:So how about we put a pin in this? \[laughter\] Because I wouldn't want to use the rest of our time to talk about this... It's something that you're very passionate about.
Jerod Santo:Oh, okay...
Gerhard Lazu:This is where I'm stopping myself, okay? ...to go too far.
Jerod Santo:Alright, I'll stop prodding...
Gerhard Lazu:\[42:11\] So yeah, there was the BuildKit issue. And the other issue - and this was like the most recent issue - was where I had a PostgreSQL container binding on the same TCP port, and then the Changelog \[42:27\] PostgreSQL container was failing to start. So while BuildKit was okay, the Tailscale key was okay, there was a collision on the TCP port, so CI couldn't run. And that was basically the last thing for me. The straw that broke Gerhard's mind. I said "Table flip. I'm fixing this." So that's why I fixed the Docker Engine and I migrated to Fly.
Jerod Santo:Once and for all.
Gerhard Lazu:I hope so.
Jerod Santo:\[laughs\] For now, and for a few people.
Gerhard Lazu:Yeah, for now. At least for the next two and a half months this is okay.
Jerod Santo:Well, while it's on my mind as we're in this kind of headspace, I wanna say one more thing I'm excited about with Fly is that I was able to, via Fly Proxy - just proxy the Postgres connection... We talked about this, how we're doing back-ups, which is -- I mean, they're doing them as well; we don't have much visibility into that. But if you want an ad-hoc back-up, you just connect with Fly Proxy and you can connect to psql directly. That feels a little weird to me, but what's great about that is now I can actually just use it as my database. I don't do that when I'm developing --
Gerhard Lazu:Don't do that...
Jerod Santo:No, I don't do it.
Gerhard Lazu:Don't do that... \[laughs\]
Jerod Santo:But I could. \[laughs\]
Gerhard Lazu:You could, but don't.
Jerod Santo:The possibility is there, and it's tantalizing. But what I can do is I can connect Postico, which is my favorite little Postgres query UI, directly to production. Because every once in a while you've just gotta munge a little bit of data and fix a thing. And with Kubernetes I could never get through whatever layers there were in order to get that done, and I think the option was like "Let's not expose it publicly to the world." But this is all just set up and working, and so I just proxy that sucker, connect Postico to production... I had this big, red background, versus green, to know "Hey, be careful. You're in production."
Gerhard Lazu:\[laughs\] Oh, my goodness me...
Jerod Santo:But I can just run arbitrary queries against our production database. And that's a best practice, isn't it?
Gerhard Lazu:Okay, so before we do that, we need to take you through a certification program, which will be "I develop in production." That's the end result. \[laughter\]
Jerod Santo:Yeah, there should be a cert for that.
Gerhard Lazu:Yeah, exactly.
Jerod Santo:A+. Isn't that what A+ gets you?
Gerhard Lazu:"I fly planes for real. I don't just work with Fly."
Jerod Santo:You should have like a two-button commit on that thing, where like you and I have to both hit the button at the same time to run the query. That would be fun.
Gerhard Lazu:That was Adam's idea. I still remember it. From deleting the DNS records, remember? That's what he said.
Jerod Santo:Oh, that was. That's right.
Gerhard Lazu:\[laughs\] Adam, you're still here.
Jerod Santo:Yeah. Here in spirit. Okay, so I just wanted to get that interval. I was thinking of it, because you were talking about connecting things to different places.
Gerhard Lazu:That was a good one.
Jerod Santo:Yeah. Okay, so we have pull request 416. Obviously, there's additional stuff to do there eventually... But good to go. Specifically, we would like to spin it up and spin it down on-demand. If that works, it'd be sweet.
Gerhard Lazu:That's right, using Fly Machines. That's right.
Jerod Santo:Yeah. Fly Machines... Cool.
Gerhard Lazu:But the thing that I want to go back to is the improvement which I mentioned in episode 50. And I think you realized why we can't run more than one instance of the Changelog app.
Jerod Santo:The clustering.
Gerhard Lazu:It's that lack of clustering... So looking at the Fly docs, how to integrate that looks really, really simple, but I haven't done it yet. So I would really want to do that. I mentioned two and a half months ago I'll do it. Maybe now I'll do it. I don't know, we'll see. It's summer, lots of holidays coming up, but still, I will get to do that.
Jerod Santo:\[45:54\] Yeah. That's something also I could maybe take a crack at, but I also have other things which I'm working on in the application space that are probably higher priority than that... Which we'd love to have ready for episode 70, which we haven't teased at all yet... But maybe now we take an opportunity to say we are working with our friend, Lars Wikman, on chapter support for our podcast episodes. And it's in-progress; no promises. We would love to have it by 70. And if we do have it by 70, we'll talk through all the details, hopefully get Lars on the show and make a big deal out of it. But that's what I've been focusing my efforts on. I probably won't get around to this... But I read that doc and I was like, "I could probably do this as well in the app, to get clustering set up." So it does look pretty straightforward. I think it'd be a good step for us to do.
Gerhard Lazu:Oh, yes.
Break:\[46:44\]
Gerhard Lazu:Just to continue with the teasing theme, episode 61, when it comes out, which is right after this one, we are talking to John and Jason from Transistor.fm.
Jerod Santo:Ah, okay...
Gerhard Lazu:And I've mentioned this to them, and they're excited to find out more.
Jerod Santo:That's cool.
Gerhard Lazu:Now, they're running Ruby on Rails, we're running Elixir in the context of this library, but they were curious to see how we do it. They use FFmpeg extensively, for other purposes, but I was saying that we're using FFmpeg today just for that.
Jerod Santo:So they would also have to ditch FFmpeg.
Gerhard Lazu:I think the bigger problem would be Ruby on Rails.
Jerod Santo:I would expect there'd already be tooling in Ruby for this. It would just be slow and memory-consumptive. Is "consumptive" a word? I would expect there to be some tools for this in Ruby land.
Gerhard Lazu:Memory-hungry.
Jerod Santo:I mean, not Ruby in general, but at this particular style pack I think it might be... Because these are large files that you're reading in the memory and modifying... I don't know, Lars can speak better to it. But now we're getting too far into the weeds on it. Yeah, that's cool... I'm a big fan of Transistor, and excited for that episode. And yeah, hopefully what we do can at least be looked at by them and integrated. Because we are gonna be editing our chapters in the CMS, and having it syndicated into the mp3 files and into the podcast feeds themselves, because the new podcasting spec has chapters built into it...
Gerhard Lazu:Sweet.
Jerod Santo:...and so you don't have to put them in your mp3, you can put them in your feed... So we're gonna do both. And then we could also have them on the website, on the episode page as well, for easy clicking around too. That's the coolest part about it. But anyways. Now I'm revealing too much.
Gerhard Lazu:No, that's all cool stuff. All that's coming. I think we touched up on it in one of the Kaizens before, where we talked about --
Jerod Santo:Why we don't --
Gerhard Lazu:It might have been 40...
Jerod Santo:Yeah.
Gerhard Lazu:\[52:12\] ...where we talked about block storage, and local volumes, and this came up in the context of some of the issues that you had to overcome to migrate from a local volume to object storage, to S3, for assets, for mp3s.
Jerod Santo:Yeah. And now everything's out of the way, and we are getting the one last thing blocking us from good chapter support; we're getting that taken care of, and then we will be good to go.
Gerhard Lazu:They'll be so cool... Because I find myself looking for specific parts of episodes that I want to link to... If there were chapters, it would be much easier to find them. And especially linking them to the transcripts. So if I know which transcript is where, it will be much easier for me to find the portion of the mp3 file where I want to link to. That's really cool. And if you add player support - now you're just blowing my mind.
Jerod Santo:Not all by 70, but these are all things we wanna do.
Gerhard Lazu:That's okay. That's, again, another reason why we won't do those improvements regularly, and just release them bit by bit.
Jerod Santo:Small steps.
Gerhard Lazu:Exactly. Exactly. For us, small steps means 2,5 months, but it still shows that with life happening, sometimes just other things become more important, and you just handle them. But we are very conscious about what is happening, because it's always in our minds. We always check the previous episodes, what we said we will do, versus what we did, and what we still want to do. So speaking of that, one thing which I still want to do is to figure out clustering. So I don't know whether we end up pairing for an hour, what that looks like, but I'm very keen by the next Kaizen to have a cluster of Changelog apps running, because I really want to have one running in London. Multi-regions, one running in Virginia... The one in Omaha - I don't think they have one, but that will be nice. That will be nice.
Jerod Santo:\[laughs\] Ohio, I bet, is probably the closest thing to me.
Gerhard Lazu:Yeah, Ohio. So again, I don't know whether they still have that region.
Jerod Santo:It looks like they have Chicago, Illinois. ORD is a Fly region. That one's very close to me.
Gerhard Lazu:Okay.
Jerod Santo:So one there, one in Singapore, one in Sydney, one in Toronto... Where else? I mean, once we're clustered, right? One in India... We're huge in Germany. Probably we'll put one there.
Gerhard Lazu:I think we should put one on every continent, to be honest... And we can use Honeycomb to see which data centers get the most traffic, which by consensus would be the most traffic
Jerod Santo:That's a good idea.
Gerhard Lazu:And then based on that, we can put one in each region. And this is like a small step towards maybe one day using SQLite. I mean, that's a crazy idea...
Jerod Santo:Yeah, because now we have, just use a better datastream in each region, right there with the app, and let Litestream do its deal... Or whatever they're cooking up next over there.
Gerhard Lazu:Yeah, pretty much. But for now, we can just basically have dynamic requests responding much quicker. One thing which I've noticed today is -- do you remember those vanity domains, and the redirects which we have, like Shipit.show, for example? So that one used to take more than 200 milliseconds for me when the request had to travel from the U.K, from London, all the way to Virginia, and back. But now, they only take 23 milliseconds. And the reason why they take so little is because there's a Fly proxy. And the Fly proxy is distributed. So I don't know exactly what magic that is, because I know that the speed of light would take more than 23 milliseconds if it had to go to Virginia today. Because that's the only place where we currently have our Changelog app instance running. So it's not hitting the app, it's hitting the proxy.
Jerod Santo:The proxy is just caching maybe?
Gerhard Lazu:\[55:57\] Must do. Because otherwise I can't explain --
Jerod Santo:How else would it know the answer?
Gerhard Lazu:Exactly. That fast. Because it must hit the app. And I can see the Fly.io IP address. Now, I didn't run an MTR to see where it is. 16 milliseconds. Somewhere in London. NTT.net. And then it's hitting the internal network, and then all I'm getting is this IP. So that's the firewall, basically. So it's my IPE, cw.net, I'm not sure who runs that, lns ltw, but it's all London12.uk.bbg.ge.entity.net. So that is the entrypoint to the datacenter wherever this is running. And there's three more hops, which is through the actual datacenter and eventually it hits the proxy IP address. And that's 6651126203. Maybe that's one for Kurt. But anyways, it was really cool to see those redirects working so quickly, because if you remember, we were saying at some point "Why don't we set up those redirects on the CDN, so they respond quicker?"
Jerod Santo:Right.
Gerhard Lazu:We don't have to do that anymore. Isn't it amazing? An improvement that you don't make, and you just find out that it's just happened?
Jerod Santo:Those are the best kind.
Gerhard Lazu:That's the best one.
Jerod Santo:Procrastinating for the win. You know, just sit around and let somebody else figure it out by happenstance. What else should we not do? What else should we strategically not do, so that other people get it done for us? Maybe that 404 thing, should I...
Gerhard Lazu:I don't know, I think we need to ask Kurt. Hey Kurt, what's your roadmap? Tell us what's coming. \[laughter\] But seriously, the whole Litestream thing, the way Ben talks about it - again, I'm just teasing what's coming in... Actually, no - what came, sorry. Episode 59. So that's already out by the time --
Jerod Santo:You're teasing me, but you're not teasing our listener, who already listened.
Gerhard Lazu:Exactly, yeah. So you already know what we talked about. He's talking about how there will be a special directory in your Fly deployment, in your Fly app. And if you use that directory, it will automatically be synchronized if you have SQLite there. They're both gonna be synchronized with the other instances. So at least that's the thinking. How it will work in practice, we'll see. But if you put your SQLite in that directory, it'll automatically be synchronized with your other app instances running on Fly. Now, that's really cool.
Jerod Santo:With zero work to do that.
Gerhard Lazu:There you go. No Litestream integration. You don't have to run the process, you don't have to configure it...
Jerod Santo:It's just like, "Put it here, and we're gonna replicate it around the world to everywhere else your app exists."
Gerhard Lazu:Exactly.
Jerod Santo:Sign me up. Except we have some Postgres-specific features that we're using, but... We will address those things as time allows. But very cool, very cool.
Gerhard Lazu:Yeah, I can see 2023 starting as an experiment, and we see how far it goes. But this is an interesting idea. And then it doesn't matter that your database is not managed. Do you even need it to be managed?
Jerod Santo:It's kind of managed.
Gerhard Lazu:There you go, it's your application.
Jerod Santo:It kind of just exists everywhere, all at once. I mean, talk about back-ups...
Gerhard Lazu:Now, that's really cool.
Jerod Santo:Yeah, that's cool. You still want snapshots though, because you could screw something up and you wanna go back, right?
Gerhard Lazu:Feature request.
Jerod Santo:\[laughter\] Get 'em in early!
Gerhard Lazu:Exactly.
Jerod Santo:That's all it is, it's a feature request. I love it.
Gerhard Lazu:Yeah. And we won't be the only ones, I'm sure of it. Anyways...
Jerod Santo:Oh, no. So it's just the first thing you think of.
Gerhard Lazu:So clustering - I think that's going to be a big deal. We'll see how far we can push PostgreSQL. I know that Adam was mentioning about Crunchy Data, using their managed PostgreSQL... That's an interesting. I would like to try it out. But again, let's see where it fits with everything else. SQLite - I'm very excited about that, especially with Litestream. It wasn't even an option until Ben joined Fly. See, that's how things happened - we migrated to Fly, Ben joined Fly, and then amazing things are being discussed. And we'll see how far it gets, but I'm excited. The other thing which is on my mind - and this is episode 58; I blame Rosemary and Rob... No. I blame -- who recommended it? Someone else recommended that episode, and I forgot. Let me check it out. Shipit.show/58. So easy. Seriously. I love it. Just that. It's just saving me from typing more.
Jerod Santo:It is. It's nice.
Gerhard Lazu:\[59:51\] And let's see, do we have -- ah, we don't have the transcript.
Jerod Santo:Not yet, it just came out.
Gerhard Lazu:Thomas Eckert.
Jerod Santo:Oh yeah, Thomas.
Gerhard Lazu:Thank you, Thomas Eckert, for the intro. Yeah, Thomas did the intro. So thank you, Thomas, for the intro. And we had some amazing comments... This is actually the episode which had the most comments in the Ship It Slack channel.
Jerod Santo:Hm. That's because everybody has opinions on how to do secrets...
Gerhard Lazu:Exactly. So we have to take a popular subject, share an unpopular or unconventional approach, and let the comments come in. That's how we do it. We've been doing it wrong all along.
Jerod Santo:Yeah. Magic.
Gerhard Lazu:So Maikel Vlasman - he was a guest for episode 56, we had him on. "DevOps teams with shared responsibilities." He mentioned that he's using Sealed Secrets, but a lot of the statements that we make in that episode, in episode 58, he agrees with them. You can see it in Slack. James Hart, he was asking how we do the LastPass integration with Kubernetes... So I referenced the link, it's there. But the one which was really interesting is Owen Valentine - thank you very much for sharing the link which shows HashiCorp Vault plugin that integrates with 1Password. That was interesting. So 1Password is not too dissimilar from LastPass. By the way, Omry Gabi, he had an amazing input... "People store application secrets in LastPass?" Yes, we do.
Jerod Santo:Yes, they do.
Gerhard Lazu:For years. And it works... Kind of. \[laughs\] Minus like a few small issues.
Jerod Santo:Yeah.
Gerhard Lazu:So if we were to migrate to 1Password, we would still need to have a password store for Vault. So where will they be persisted. And we can use various integrations, but 1Password is one of them. Do you use 1Password, Jerod?
Jerod Santo:Adam does. I do not. He's tried to get me on the 1Password team, or whatever their -- Pro, or Business, or I don't know what their plan is called... And I'm open to it, but I was always like -- I'm spread across so many at this point; why do I want to add yet another? But if we're gonna consolidate, I'm open to that.
Gerhard Lazu:Well, one thing which I haven't shared is that I've been experimenting with 1Password for about 3-4 months now. So I've switched from LastPass to 1Password. I still have LastPass around, but I'm using it less and less. And I know that 1Password works pretty well... So maybe we can take this opportunity to consolidate at 1Password, move everything across, and with HashiCorp Vault do that integration. So we can have the best of all worlds. We can have the LastPass CLI locally, we can have the browser extension, we can have the HashiCorp Vault integration for the application... That would be really cool.
Jerod Santo:Did you say LastPass CLI, or did you mean 1Password CLI?
Gerhard Lazu:I meant 1Password CLI.
Jerod Santo:Okay, because you confused me there.
Gerhard Lazu:That's what I meant.
Jerod Santo:Okay. So they have a CLI, just like LastPass does.
Gerhard Lazu:They do.
Jerod Santo:So you can have the best of all worlds.
Gerhard Lazu:Exactly.
Jerod Santo:1Password CLI, HashiCorp Vault...
Gerhard Lazu:Yeah. HashiCorp Vault, with integration to 1Password Connect. Yes.
Jerod Santo:And we can share passwords amongst ourselves, or you can also have your personal 1Password stuff... It's like one big, happy, secure family.
Gerhard Lazu:\[01:03:08.10\] It sounds too good. We need to try it out.
Jerod Santo:Okay, I'm open to that.
Gerhard Lazu:I think that's cool. I'm very excited about that... And not to mention I already use it, so... The more difficult question is which plan do we add people on? And I think, realistically, we'll join multiple plans, right? I think if Adam has his, we may join that, especially if Changelog has one... And then we can store those secrets there.
Jerod Santo:Yeah. We would have a Changelog plan of some kind. And if it can be multiples, then...
Gerhard Lazu:This is an action item for Adam. This was a retro... This is an action item for Adam. Adam, you're listening, right?
Jerod Santo:Right.
Gerhard Lazu:Add myself and Jerod to that plan, so that we can start using it. That would be really cool.
Jerod Santo:Alright. Anything else?
Gerhard Lazu:Well, was there enough laughter? What do you think?
Jerod Santo:Should we work in a little bit more laughter before we go?
Gerhard Lazu:Maybe...
Jerod Santo:I've got a few laughs...
Gerhard Lazu:I don't know, it depends... It's gotta be funny enough, I don't know... \[laughs\]
Jerod Santo:Well, it's tough to just demand more laughter right here at the end... But if you have any dad jokes... Or dirty jokes. It sounds like they're more your style... \[laughs\] Maybe you can squeeze one in here.
Gerhard Lazu:Okay, okay... Noah told me a good joke, and he said "Is that a dad joke?" And I said "Yes, it's a dad joke." But I forgot the joke. So hang on, I have to ask Noah...
Jerod Santo:Okay...
Gerhard Lazu:I have to go and ask him, "Hey, Noah, what was the joke?" I forgot that. I mean, this is actually a question for you, for the listener. Do you think there were enough jokes? How do we improve the Kaizen?
Jerod Santo:How many jokes per Kaizen are you expecting? What would be your threshold for happiness, and then pure joy?
Gerhard Lazu:And the ratio to technical content. Like, was there enough technical content? And how much do you miss Adam? Because if you don't miss him, I think this is the new format... \[laughter\]
Jerod Santo:It's much more efficient. \[laughs\]
Gerhard Lazu:Yeah.
Jerod Santo:So yeah, let us know.
Gerhard Lazu:That's how you end up with Boaty McBoatface. You ask the people what they want. \[laughter\]
Jerod Santo:Boaty McBoatface... Oh, nice. Alright, people, let us know,
Gerhard Lazu:Any last important takeaways, other than happy 4th of July?
Jerod Santo:Shipit.show.
Gerhard Lazu:Shipit.show. That's a great one. I love that. Let's end on that. Shipit.show. Never s\*\*t show. Never. Okay?
Jerod Santo:Never! \[laughs\] How dare you even bring that up again?!
Gerhard Lazu:No. No, no, no. It's funny, right? We wanted more jokes.
Jerod Santo:That's true.
Gerhard Lazu:Okay, so here's a crazy idea... If someone registers s\*\*tshow, and they just redirect to this... I hope that doesn't happen.
Jerod Santo:Why are you giving them ideas? \[laughs\]
Gerhard Lazu:We just said more jokes.
Jerod Santo:You're teaching people how to troll us. This is not the way it's supposed to work,
Gerhard Lazu:Alright, I think this is a great place to stop. On that bombshell, as someone famous that I very much like and admire... It's time to end. See you in two-and-a-half months, everyone. Have a great one, keep kaizening. See y'all.
Jerod Santo:Kaizen! See ya.