DevOps Sushi

That's when like my fascination with computers and automation especially started. And that escalated to me having like renting servers in Germany and having hundreds of accounts playing the games for me. And then selling that in-game currency for a little bit of extra money. I could by no means live off it, but it was just so cool to know that I had this army of bots. playing for me as I was doing my work. So in that process, I learned Python

coding, Linux. Everything was running running on Linux, and I was doing all of this in my free time and I was always like, well what if I could make this my job? Like not automated games specifically, but like working with infrastructure and working with code, automation. And I tried a few times, but it was always like, well, you don't have a CS degree and you don't have experience in the field. And this was when I was living in Norway.

So that was very tough to get into, but at some point I was again reaching that point in my nursing career where I felt this is not going to be the thing either. And then I just went, I asked myself the question: What do I want to do? And I answered that by saying: what do I like to do in my free time? And that was what came out. And then to get back to your question, I started talking to people who I knew in the IT industry. And one of them said, I think DevOps is your thing.

I think that's what you're describing to me. You're what you like about code and automation. I think DevOps is the thing. And that put me on the track on DevOps. And as soon as I heard that word for the first time, I was basically sold. Okay. So as you were discovering DevOps,

as you're getting more and more into it, what are the fun things that you didn't expect to begin with, but you really enjoyed? You've been doing this for four years now, five years, four years? I'm going into my fourth year now, yeah. I really love to learn, and learn new skills, new technologies. And I hadn't fully appreciated when I started out that this field, or IT in general, is just such a constantly changing field.

And you're basically required to tinker in the evenings just to stay up to date. It gives this constant intellectual stimulation for me and something to keep up with and to share about. Which is something that I didn't realize in the beginning. I thought, okay, I have a skill set I need to learn. I learn it. And then that's what I'm going to do. But that's one thing that really surprised me and that I still enjoy a lot.

What about the hardest part? I did have this vision that I would sit behind my computer and code and tinker with what I like to do. And that was it. And I heavily underestimated the soft skills part of it. So that in DevOps, you're all about breaking down silos and helping teams to reach their goals as a DevOps Engineer. And I didn't realize how much interpersonal communication and meetings and all of that is involved with that.

It's not necessarily the hardest part about it, but that was something that I had to like readjust myself to. But then my nursing skills, and my project management skills like really came to shine there. So that's, I think, part of my success as a DevOps Engineer. That background that I could apply those skills in those environments. And yeah, another hard part about DevOps is that it's sometimes hard for me to realize that not everybody is enthusiastic about these things as I am.

So if you come into a company and yeah, we're all about Kubernetes now. Let's start. Let's do everything Kubernetes. And this is why. And then it's like: hold on there cowboy. You said the K word. I said the K word. Get out of here. Mischa will no longer join this meeting. He said the K word. That can be a bit challenging. And that's one thing you have to learn if you're passionate about something that not everybody is going to be as passionate about as you are. What does it mean

"DevOps - breaking down silos" in practice? Do you have a story or an example to give that illustrates that concept? We were basically the cloud platform team and then we had development teams relying on us running that platform logically. What would happen is that we would get a

Okay. Well, it can wait a day, okay. Let's plan a meeting, and let's go over it together. And I would, I would, of course, look into it myself and see if there was like something immediately going wrong with the database itself. But then I would actually jump on a call with the developer who sent in this ticket, and rather than just me fixing it, I would do it together with him.

And not necessarily because I didn't know what to do, but I was actually showing genuine interest in what is the application doing, and how does it work. And when you start doing that, then the people, what they're working on, they start to get very enthusiastic about it. And in my experience, then you get into a bit of a different mindset together instead of like, well, there's this error and we're fixing it.

It becomes more of a personal project that you're a fun thing that you're solving together. And when you do that, you become more relaxed. And then in my case, we found the solution much faster. And more often than not, developers were pointing out stuff, like we were screen sharing, and we were in the monitoring. And I thought that I had seen that I had analyzed the problem and I thought I had it,

Yeah. The second step is accepting the ticket, and going with it. Don't do that. Try to have the conversation. Try to work on it together, and be open to working together. Full stop. Yes. We are in this together. It's not me, and I'll figure it out, and you'll just get the end result. We will go through it together. Bonus points if you can record it. One of the best things which I've done is recording internal meetings like this, because we are a remote team.

Jump on Riverside, or any remote software exactly like this. Do the recording, do the screen sharing, do some minimal editing, and then you have a video of how you solved the problem together. Yes. Post it internally and it's a reference for example: what does it mean to be on call? How do you handle an incident? What happens when you take an application into production? If a migration, if a database migration fails, how do you tackle it? How do you build preview environments?

Why do you need preview environments? All sorts of questions like these that come up, build a knowledge base, a wiki of some sort, A Zettelkasten. It's a system for wikis, but still something that captures that knowledge. And it's not different people doing things in a corner. It's people coming together, recording that, and then sharing that knowledge. And with all the tech that we have nowadays, starting with AI, transcription, this is easier than ever. You still have to do it. Totally agree.

And it's one of the main emphases I have when I work somewhere is usually when I jump on a team, the documentation sucks. And I always make it a point of improving that from day one. Like there's always the onboarding process where you can't really do much the first couple of days. I always start with reading the documentation, asking questions and improving it already as I go along. That's amazing. Document first, automate second. But always document.

I used to do things wrong for many, many years. I used to like automation. There's this Makefile. How do you mean? It's like self-explanatory. And people would tell me, no, where's the docs? Who needs docs when everything is here in the Makefile? It took me a while, but I finally got it. Documentation first, automation second.

Are there any hard-earned lessons, any hard-earned DevOps lessons that you would like to share with us? It's really important to keep an online presence and keep that well up to date. So the moment you start, like you can always lose your job. And the moment you lose your job and you haven't updated your LinkedIn profile for three years, or even not having a LinkedIn profile at all, is one of the biggest mistakes you can make as an engineer, in my opinion, in this day and age.

And I don't necessarily like that it's the case. Like I still am the guy when I'm on the street and I'm taking a picture with my cell phone. I feel weird. It feels still like this abnormal thing to do to me. Like I wasn't a social media person at all. But once I recognize the potential it has and the importance it has these days for your career, then I think this is one hard-earned lesson that I would like to give everybody to share your journeys, even if it's just one post a week.

But be active online. It will only only give you benefits. Build your personal brand. Whatever that is, whatever works for you. If it's TikTok, if it's Reels, if it's Instagram, it doesn't really matter. Twitter, Blue Sky, say Twitter... X, shall I say or Xitter. I've heard that's like a new thing, which is like catching apparently. Xitter, that's a funny one. In Mandarin, it actually means something else.

So anyway, I just read recently, the point is whatever works for you, videos, podcasts - do that. And if it's too much to start one, or maintain one, join one. There's so many around. Yes. So do whatever you can do to put good content out there. Do that. Personal blog. Does not matter what it is as long as you do that, because it's like compound interest. You're putting in the bank of your personal brand. It's you. Be who you are and show the world. Exactly.

If you have a stack of 200 CVs, and there's one person there with a thousand followers and a personal brand, he literally has a better chance of getting the job. Again, I don't necessarily like it this way, but believe me, it is this way. Another hard-earned lesson that you've shared, and I picked it up from the various YouTube videos that you posted and blog posts, is: you waited too long to start a home lab. Yeah. I'm an academic-oriented person. I studied at university.

I love reading books and getting into theory and writing notes and getting more and more and more information in my head. But when I started learning Linux seriously, professionally, I'm ashamed to say this, but I wasn't entering the commands into the command line for months. I was just reading about them and I said, "Oh yeah, okay, well that's how the file system works." Great. But when I sat down and actually tried to do it the first time, it wasn't that great.

It hadn't stuck in my mind at all. One huge mistake I made that I wasn't applying what I was learning in practice, and you really need to do that in order to make things stick. And one of the best ways to do that is to start a home lab in whatever shape or form that is. Anyone can do it with any hardware. That's it. It's becoming more and more accessible and you will learn so much. So just go and learn. Have fun. Explore. Tinker.

Exactly. Yeah, I also always used to associate this word "home lab" with these huge server racks with blinking lights. That's what you see on YouTube. I never understood that you could actually do this on cheaper hardware or on something that you already have. So my students, I always say, Okay, go to your closet. There's 99% sure there is one old laptop in there with maybe four to eight gigabytes of RAM. Install Ubuntu server on it and you have a home lab.

You have a learning environment where you can learn enough skills to get a job. That's literally all the hardware you need to get a job.

I went through every single article on your blog. It's amazing. So thank you first of all for taking the time to write them, and then to share them. And out of all of them, there's one that resonated with me the most. Do you want to guess which one? My favourite one is: "I'm in love with my work" So I'm not sure if that's the one, or it's "The power of writing". It's either that one or the other one. Okay, so I'm going to read a quote and you'll recognize it, I think, after the second sentence.

Once you decide on your occupation, you must immerse yourself in your work. Yeah, I know. You have to fall in love with your work. Never complain about your job. You must dedicate your life to mastering your skill. That's the secret of success and it is the key to being regarded honorably. Who said that? Who wrote that? Jiro. Jiro. Who's Jiro? Tell us about Jiro. Jiro. Oh, Jiro is the protagonist or the main character of the documentary "Jiro Dreams of Sushi."

And when I first saw this documentary, I did not appreciate how deep this actually is. But Jiro is a sushi chef who operates out of a subway station in this tiny little restaurant with like four seats in it. And he is the master of sushi of the entire world. Like there's nobody better than him. And I know nothing about sushi. I have no interest in sushi. But this documentary is the story of him like committing to his craft of becoming a sushi master. And what they call in Japan a "shokunin".

Someone who is completely dedicated to their craft and just doing the same thing over and over and over and over again and making it perfect and even more perfect the next day. That is the shortest description I could give you. That's a great one. So this restaurant is in Tokyo, by the way, Japan. So if you're there, I forget the name of the underground station, but you will find it because there's just one like this in the entire world actually. It's a three Michelin star restaurant.

Only one of its kind in the world. And some say Jiro is the best sushi chef that has ever lived. Now, this documentary was recorded, it was actually was it came out in 2011. Jiro was 85 years old. And he's been doing this. He's been doing sushi every day since he was 19. Yeah. Today, Jiro turns 99. I say today when we're recording this actually maybe a few weeks ago, he turned 99. I'm wondering, is he still making sushi? Well, the last time I checked, that's a couple of years ago.

Then he was also in his 90s and he was still there every day in his restaurant. That's amazing. I didn't realize this, but in the Japanese culture, apparently retired is not that well understood. No. When we first talked, I had a title in mind for our conversation. Do you remember what that was? The Note Taking Santa Claus. The Note Taking Santa Claus. That's it. Okay, we will get to that in a minute. I hope or a few minutes we'll see.

But after watching "Jiro Dreams of Sushi", and reading all your blog, I came up with a new title proposal. Let's see if it's as good as DevOps Sushi. I was thinking: "The Bear that Dreams of DevOps". The Bear that Dreams of DevOps. Wow. Okay. Why is the bear an important thing for you? Because I know it is, but why is it? Because people listening to us, they won't know. You really have read all of my blog. I'm so honored that someone...

This also is featured on the blog and why I chose the logo at the time that I had. And Mischa is a Russian name that means"little bear". So I'm conveniently ignoring the little part here, but it might not seem on the camera, but I'm Dutch and I'm almost two meters tall. So I'm a pretty big guy. And in Norwegian, they have this word called Bamse, which also means teddy bear. But they usually describe that with for long, big hairy guys.

And well, if you're watching this on a video, you'll see the connection there. For our listeners, imagine Thor. Wait, let me get my hammer. Exactly. Yeah. That's the only thing missing from the picture. Yeah...

So another thing which I watched over the weekend on top of "Jiro Dreams of Sushi" which is a great movie, 10 out of 10, highly recommended, is "Why I use Kubernetes for my Homelab." A video that you posted not that long ago. We'll put a link in the show notes. And I'm wondering if you could take us back to the day that you decided to use Kubernetes for your home lab. I think it's a very interesting choice. How did it start?

That was when I was working in my first DevOps role, and we were using Ansible for everything. and we were using Ansible for everything. And they had built this super, actually very impressive system of deploying Docker servers and then having three Docker servers, having running the same container and then putting them there. and then putting them there. But when one of those containers would die, then we would have to then we would have to go in and fix it, right?

Because you're limited by the tooling, even though there's probably ways of doing it with Ansible alone. That was my reality of the day, of the situation I was working in. And when I then for the first time created a Kubernetes cluster, and I created a deployment with three replicas, and I killed one of those containers, and I killed one of those containers and then it came up automatically, I was sold.

And then when I did a rolling update, where I would update the image tag, and then it would go one by one, and then update the containers. And if something would go wrong, it would roll it back. That's when I fell totally in love with it. And I knew I could not go back. and I knew I could not go back. For me, that was the moment where I thought, this is what I want to work with. this is what I want to work with.

When I then decided to do my home lab, to create a home lab, like it wasn't even a thought, like it wasn't even a thought, I didn't ever even consider anything else. It was just, yeah, I'm specializing in Kubernetes. specializing in Kubernetes. Of course we need to run Of course, we need to run Kubernetes in my home lab. Kubernetes in my home lab. Even though it's completely overkill, and it makes no sense, it had to be done on Kubernetes. Well, actually, I think it makes a lot of sense.

And you talk about this in your video as well. Why it makes sense, and why it is important. And the majority would react that way. It's overkill. Of course it's overkill. Why not use Docker? You started with K3S. Why? Part of me wanted to go the kube-adm way, Part of me wanted to go the Kubeadm way and just do it properly. and just do it properly.

Even kube-adm is a bit easier than doing it the hard way, like compiling the binaries yourself, etc. binaries yourself, etc. Still a goal that I have for maybe next year. have for maybe next year.

I went for K3s because I recognized I went for K3S because I recognized that the sooner I got up and running, and actually working and actually working with the Kubernetes API, and feeling that I had and feeling that I had something stable to work with, the quicker I did that, the more I would learn, the more I would learn, the quicker I would start learning. the quicker I would start learning. So I think it might felt like a shortcut at that time.

but I thought long-term and I thought, But I thought long-term, and I thought, well, with kube-adm, like, like running your own CNI running your own CNI, and having to update your certificates, like, it will work in the beginning, but if something breaks, then I'm just sitting there tinkering for hours. And I wanted something that was a little bit more thought through, and K3S was a very good option for that. Why not Kind?

I had a few Linux machines I had a few Linux machines, I had a few Linux machines and I really like this idea of that k3s does, where you install the binary, and then you can get a token, and then you can add another node to your cluster. And I actually, you can of course do this with Kind as well.

But I really like this idea of maintaining a Linux machine, maintaining a Linux machine where you install something on, where you install something on, but you still have the possibility of SSHing into it, and you still are a Linux administrator at that point, right? And I really like that balance, that I could have those Linux machines running Linux, I could tinker with it, I could still run other things on that machine as well, alongside K3S, and then just expand my home lab as I went along.

I really like that concept. Okay, so you mentioned SSH, which is interesting, because then you switch to Talos, because then you switched to Talos and Talos prides itself on not having SSH. Yes. Why did you switch to Talos? Well, I mean, that was that stage in my home lab. And then later, I was running applications that I was thinking, well, this is actually running very stable, and my home lab has now escalated to a point, where I don't want to live without it anymore.

It's actually containing applications that I want to keep running 24 seven, that I want to have accessible from the internet. And then the question of security comes around. And you need to, when you start exposing things to the internet, you need to really know what you're doing. It's a very dangerous situation. If you don't know what you're doing, and you're just opening ports and letting people into your cluster, they could break out of your containers and do bad things.

So at that point, I felt, okay, I want to have things more secure. And then Talos came on my radar, which is production grade security out of the box. And it wasn't necessarily because I felt incapable of securing my own cluster. It was more a combination of learning a new technology and also having this peace of mind, knowing that what I was doing was actually secure. So then I switched to Talos and yeah, I'm never going back. I think it's been amazing so far.

If someone wants to get really good at Kubernetes and start a homelab today. Yeah. How would you recommend that they do that? I'm currently in the process of building a course for this, the Kubernetes Homelab. It's 60% done and people are responding really well to it in the community. They love it. And the approach I give there is, first I have my Kubernetes Fundamentals course where they run Rancher Desktop on their own machine and then to learn Kubernetes.

And then the next step is to actually get a laptop, a Raspberry Pi or some sort of dedicated hardware and then run k3s on it. That is the approach that I advise to do. So first learn it in a local setting and then move to dedicated hardware, use k3s and learn GitOps as soon as possible. So where could people find this course?

The course is currently available in my community, KubeCraft that I started at the beginning of this year, where I invite engineers to come in and engage with each other and where I also have my note-taking material and my Kubernetes courses. Obviously we'll add a link in the show notes. I had a quick look. I haven't joined the community yet. I'll put yet there. I've seen that it's on skool.com with a k. Can you tell us a little bit about that?

Well, maybe I can just briefly describe why I even started doing this. As a YouTuber, I was just making videos because I like sharing knowledge and then at some point people were actually telling me that: "Whoa, we really like your approach to explaining things and how you approach things. Don't you have a course on X or Y?" And when I realized that my audience started asking me for this,

And then I researched the platforms and of course there's Udemy, there's Kajabi, etc. But what Skool really emphasizes is combining education with community. And when I heard that sentence, I was like: "Yep, this is it." Because I don't just want people to buy a course of me and then that's it.

I mean, that's fine, but it's much better if you can create a group of people who are gathering around their interests for those courses and then can get support from me, but more importantly from the collective knowledge of that community. They can actually draw from each other and discuss tactics with each other. And that is proven to be very powerful so far. So this community, when you started earlier this year, started at zero. Where is it today? We are currently at 401 members.

Wow, so just past 400, okay. So it'd be interesting to see how many members there will be by the time this comes out. And you'll have the delta to see how that grows. I also know that obviously this is a paid community because of the value that you get. But there's also a free community that you have also on Skool. The free community now has 2000 members. And I only started that a month ago. Wow! So that has been very successful in that sense.

Well, I do ask, I charge a monthly price for membership of the premium community. The services that I provide for KubeCraft.

We will make sure to add both links in the show notes so people can go and check them both out to see where they want to start and what they want to do next. What would you say is the main reason that people go from free to the premium one? I think there are two kinds of people in the world in terms of employment. There is the guy who shows up at work, does his work, goes home, and then plays video games or does whatever he wants and then he shows back at work again.

And then there's the guy who is entering the stand-up who excitedly talks about a new feature that came out on Talos or he managed to get this self-hosted app up and running in his homelab. And he talks about that. And I'm obviously the second guy who likes to tinker with what he does in his free time as well. Well, imagine putting 400 people like that together, all sharing what they're discovering and what they're learning. And what you get from that is a sense of belonging.

I was never able to find this place of geeks and nerds sharing homelab stuff and learning Kubernetes and learning my approaches to note-taking and personal branding and doing that together. And I really found my tribe in this. I created something that wasn't there and I really feel at home there. And there is, we have calls there as well a few times a week where you can ask me questions directly on how I approach things. But what's even better, I don't try to be the guru of the community.

I'm all about letting the community answering things. Because I can only know so much. I only have this experience. When you put 10 experienced engineers in a call and when you come in there, maybe someone who wants to transition to DevOps and you're in that meeting and you're saying, "Oh, guys, I have this problem in my homelab." or "At work", or, "I don't know what my next certification should be."

If you have all that experience sitting there in a meeting, you're going to get the answers and you're going to get good answers. And it's amazing. I'm amazed how well it all works and how much progress people are making.

Coming back to the technical side, we have a couple more things to go through before I'm very keen to start screen sharing, and showing our homelab, because that's what this is about. Talking about it, but then also showing what that means in practice, which I'm very excited about. What was the hardest issue that you've hit with Talos? When I set up my Talos cluster, I had baked a custom image. There's the image builder, and then you can add certain drivers that you need.

But I did that, and I apparently didn't document that properly, because I need the iSCSI tools for my Synology to be able to provision storage on the cluster. And I had updated my Kubernetes cluster with the Talos update, so new Talos images are loaded onto the nodes. And all of a sudden, all of my storage was broken, and I can be honest, I panicked. Fortunately, everything is backed up to the cloud. But that was a big problem at that point.

And then after a lot of debugging, thinking it was Synology pods on the cluster itself that maybe got updated or weren't compatible, I finally realized, oh yeah, I had that iSCSI driver that was missing. So that was the biggest one. Yeah. Anything that happened in the recent update that you haven't posted yet? I ran into something where I would initiate a reboot of the node, and then in the CLI, it will then, say, now waiting for the node to come back up.

And then I could see in my closet where the nodes are standing, it was up. It was up and running. But it wouldn't work. I would have to turn it off and on again, and then it would reboot properly. Like, I decided not to spend too much time into why that was. At least I knew how to fix it. But that was an interesting one that I had this time. So this is a tip from the experts. Turn it off and on again, and it will fix itself. That's one. Which version did you upgrade from and to?

I went from Talos version 1.7.5 to 1.8.2. That's it. Yeah. I assumed this was it. From 1.7 to 1.8. Oh, really? I did went from 1.7.5 to 1.7.7. So I was a good boy, and I followed the latest patch upgrade, and then went to the next minor version. But you had the same problem too, apparently. So I don't do in-place upgrades. It's one of my hard rules. I never do that. Actually, my clusters, they have a date on the cluster name, which tells me exactly when that cluster was created.

And I never upgrade minors. I rarely upgrade patches. I just need to know when. Even Kubernetes, sometimes I never upgrade, for example, from one Kubernetes minor to another Kubernetes minor, even though most experiences are good. The problem is everything else that you have running in the cluster is very difficult to reason about the combination of the different tools that you have, because it's not just Kubernetes, not just the operating system. There's also everything else.

What I found is that if you do in-place upgrades, you're risking it. The more upgrades that work, the more likely you will hit a bug at some point where you wish you hadn't done that. Yeah. If you don't do in-place upgrades, it forces you to encode the system in a certain way. It forces you to restore from backups. It forces you to have a certain approach, which is operationally more mature.

It's a bit harder, but you can sleep safely at night knowing that you can restore everything, even if it was to burn down, because everything is configured, everything is backed up. You just point to a new cluster, everything gets restored or you create a new cluster, you give it a new config and it boots as a new cluster. Most of my workloads, they have built in a way to restore themselves.

What that means is that in the init container, when a workload starts, it looks if it's the first time that it's booted, it's like it's a bootstrap phase. And if it's a bootstrap phase, it will go and restore from an S3 compatible API. Nice. In my case, I use two. I use Backblaze B2 and Cloudflare R2. And I do this via rclone. So rclone is the first thing that restores, even MySQL.

You'd be surprised how well this stuff works for even for large data, because you're pulling down from object storage that is supposed to be performant. Backblaze is not as performant as Cloudflare R2, or at least in my experience, but I have both because you don't want to put your eggs in one basket, especially backups.

So I have two of everything, another rule that I have. The other one is don't do in-place upgrades. It's not worth it. So when you do an upgrade, then you would actually just spin up an entirely new cluster? Okay. So also just like completely new, well, not new, but like different hardware, spin up a new cluster, and then you decommission the old one, and that becomes your staging cluster. I'm thinking of it blue-green, but long-term blue-green. I'm always between blue and green.

And it removes all the pressure from having to do the upgrades straight away. It'll either work or not work. No, I'm setting things up. It's almost like buying a new car, but in this case, it's more practical because it's not a car. You do need to have a bit more hardware, but you have that anyway, because you have generations of hardware. You have your old laptop and the older laptop and whatever. So that's my case as well. Also, very controversial, my clusters are single-node. Ooh, spicy.

I know, right? Yeah, you can get some pretty big single-node instances. And unless you really understand your networking and can deal with etcd, and you can do uneven numbers, knowing how Raft works and how leader election works and all of that, you need three, five, seven, you're pushing it a bit. If you go beyond nine, you're basically like in a whole new world. I think very few have more than nine nodes in their homelabs. But that should be the absolute limit for many reasons.

These systems are so advanced that they will continuously converge, and you can have problems and not even know that you're having problems. That's how refined they're getting. And usually when you get into DNS and if you get into network latencies and switch upgrades, and you have to deal with all these scenarios. Single-node, you don't. In that metric, I am currently at four, but I had five for a particular reason. But you have a single control plane? Currently, yes. Right.

So that's slightly different. So single control plane, that's almost like the middle ground. So the control plane is not HA. In my case, my control plane is not HA, but all my workloads fit on the control plane node. Yeah, I see. And then I have two right now, and then basically one is blue, one is green. So I have two home labs, and I just switch between them. And it takes me months to go from one to the other. So nothing is rushed. I can experiment, I can try things out.

And if one was to break, I always have like a spare that I can go back to. That's the other thing. I used to have a staging cluster and that ran exactly the same. And I would just then update the GitOps for the production one if everything went right. However, I actually started needing to expand my production cluster with extra nodes. And I cannibalized some from my staging. And so that one's not operational anymore.

And to give you an alternative perspective on not doing in-place upgrades, I found that it makes me think about my systems to have them more resilient and easier to restore. And I'm not in the stage where I have auto restore like you, but what I emphasize a lot on my current cluster is I don't want to have anything persistent outside of databases. So my databases are running in with the EDB operator, Enterprise DB. And then using Barman, I'm also storing those in object storage.

And that has been a really good exercise like four times already. I lost all of my databases because I was a bit too experimental. Tinkering. You were tinkering. Yeah, it happens. That's how you know that you're tinkering the right way. When you break stuff, you're pushing the limits. So I've actually gained a lot of confidence in my setup now that I can actually just delete the whole thing and then restore it from object storage because everything is in databases.

And anyone who's listening to this databases on Kubernetes is easier than you think. And it works extremely well with the Barman object store. It's that. I'm so, so excited about it. It's super cool. In my last Homelab presentation, which I gave at Taloscon, I use CloudNativePG, but also Barman and object storage. And that's exactly how I migrated from homelab to production.

Because in this case on homelab, I was running a workload that was backing out via Barman to R2, Cloudflare R2 in this case. And when I set up production, I was restoring everything exactly the same way. Would you recommend iscsi-tools? Would you recommend it? Like does it work well, the extension in Talos and like the whole setup? Because I'm thinking of setting up the same thing. I don't have that yet. But would you recommend it?

Like I can't speak to iSCSI tools specifically because I just run that and it works. So I have a Synology NAS, and then I use the iSCSI tools in Talos that needs to be installed. And then there is a CSI driver for Synology for Kubernetes. And that allows me to provision persistent volumes on my Synology NAS. And I must say I'm really impressed how well that works.

I know that you started with Argo CD, but you're currently using Flux. Yes. Why? I work with Argo CD at work. And when I first learned Kubernetes, I was super excited about Argo with UI. And still, it's like a fantastic tool. That was my first GitOps exposure. But then I'm a specialist in Microsoft Azure, and I'm also a Microsoft MVP. So I focus a lot on Azure Kubernetes Service.

And in terms of enterprise-grade Kubernetes, I like to, like, as I mentioned before, I like to explore and experiment, but I also need to focus myself. If not, I get too broad and I just get overwhelmed. So in terms of production-grade Kubernetes, I decided to focus on that ecosystem for a while, which has been a good decision in my case. And the native GitOps offering in AKS uses Flux, which I was very surprised by when I first tried that out.

And I noticed at work as well that I was kind of losing my CLI ninja skills in terms of kubectl because I was deploying so much through Argo CD, which was, I saw as a problem. And what I like with Flux is that it's much more CLI focused. And of course, like, usually you're not actually interacting with the cluster in that way. But in terms of my homelab experience, I want to be in there and tinkering and just talking to my cluster directly.

And then the Flux CLI felt very intuitive to me, even though Argo CD also has a CLI, I'm aware of that. But the way how Flux does it with the custom resources such as Helm releases and Kustomizations, and being able to check those resources and Git repos, all of that, I really like the way how that is set up. And also, I completely love the way how Flux bootstraps onto clusters, how you can make your cluster like a self-feeding machine. It's just super awesome. Okay, interesting.

How was the migration from Argo CD to Flux? Because you had like a bunch of things all declared in Argo CD, I imagine. Was it straightforward to migrate to Flux? At that point, when I was using Argo CD, I was getting very Helm focused. And at some point, I realized, well, not everything that I want to run has Helm charts readily available to me. So I need to dig deeper into this. Then at the same time, I was discovering Flux. And there is a lot of emphasis on Kustomize.

Basically, everything works with Kustomizations. That was basically an incentive to finally learn Kustomize. I had been putting it off for the longest time. I just tore everything down and rewrote everything in Kustomize, which has been a really good experience in terms of my development. And Kubernetes learning. When Rio mentioned you,

he caught my attention. And that was him mentioning how you spent nine years in Norway. Roaming the wilderness. Yeah. Reconnecting with yourself, reconnecting with the nature and remembering what is really important. Well, I must say, like, it wasn't that I was doing that full time, right? I was just living a normal life and working.

But basically, if I wasn't like tinkering with computers during the evenings in the week, every holiday, every weekend, every opportunity I got was spent being in nature. was spent being in nature. This will sound so cliché. But one of the reasons that got me into this was the movie "Into the Wild". And just that phrase, "into the wild," I have since become less of a fan of Mr. Christopher McCandless, who actually turned out to be rather stupid.

But when I was young, I didn't actually appreciate his mistakes. Let's keep it that way. But that idea of being out and being self-reliant, that really applied to me, and spoke to me. And... I have since come to realize that that is also not possible. So that's also why I was... that I went back, basically. But the idea of the experience of just strapping on a backpack and driving your car somewhere with a vague idea of what you're going to do.

And then I would just bring a sack of oatmeal and my fishing pole. And I have since turned plant based, but then I was still killing animals and eating them. But that did allow me to be out there alone on a self-reliant matter. And when you do that, then you... No phone, alone, in the wilderness, no people. First of all, you become really aware what you're doing, because if you make a wrong step and you fall, you break your leg. And that is almost certainly going to be your death.

So you have to be... You have to know what you're doing. And you have to be really aware of what you're doing. And when you're out there not talking to people, just there with your own thoughts, things emerge that would not emerge in the day-to-day stress & busyness that we all experience. And I've had some very deep philosophical insights and discoveries about my own life. Just sitting there, by the water, staring into the fire that I built myself... for hours on end.

And you'll be amazed what the mind then brings up to you. So that's an indication of what that was like. Do you still do that? I mean, do you still make time for going out there and being with yourself? No phone, no people. I think an extension of this whole wilderness roaming that I did was that I got really into meditation and Buddhist meditation specifically. And I have now found ways of getting that experience decoupled from being in nature that way. So I'm a very consistent meditator.

I meditate for several hours on end sometimes. And I moved back to the Netherlands, and here nature is much more difficult to find. It's here, I have nice areas around me, but I do really miss being in the forest and just knowing that you are alone there. And the idea that you cannot meet anyone, or it's not likely that you'll meet everyone. Like here in the Netherlands, there's always someone around you. I did go back to Norway a couple-- one year ago.

I spent three weeks driving around and just roaming and having the time of my life. Last year, I didn't do it, and now it's really itching to go back. So I try. I know that each of us has a place where we feel at home. And it's rarely where home is. It's usually like somewhere maybe nearby, or it can be a different country for sure. But when you're there, you feel like you're home, and it's very hard to explain because everything is the way it's supposed to be.

Everything smells right, it sounds right, you feel very at ease with yourself. Is Norway that place for you? That's a deep question. That's one that I... Like Norway is extremely close to my heart. Like one experience that I had was when I moved back home, I hadn't visited for two years. And then last year, I went there and I drove off the boat and it felt like it didn't feel special. That was interesting. It felt like, oh, I'm here again. So in that sense, yes, it does feel like home.

But I'm also really starting to like my little boring life in just a one-room apartment and focusing on my and focusing on my business and the technology and having a more routine life in that sense. I think this is potentially the most important takeaway from our conversation. Going outside. Slowing down. You don't necessarily have to go outside, but usually it helps. Slowing down your mind, and... finding your happy place. It's usually in your mind. The happy place is in your mind, by the way.

Outside, walking, being... free from distractions usually helps that surface. But ultimately, you still have to do the work and you still have to settle your mind and go through all the steps. But it's definitely out there. Everyone has it. Yes. And nature is a great facilitator for that. And I think in my case, nature was the gateway to inner peace. Nature was the place where I could disconnect that way and sit by the fire and experience that. And still, of course, still nature is that for me.

Last year I was there, I walked 150 kilometers in the mountains and just alone. It makes you feel very small when you're in those big mountains and a thunderstorm breaks out and it puts so many things into perspective. And how powerful those forces are and how little we are in comparison to that. That is also very healthy for me to have that humbling experience every once in a while.

That's it. And when you come back to whatever you normally do, you'll be so much more productive and things will flow a lot better because your mind took a break. Absolutely. Okay. Well, Mischa, thank you very much for joining me today. It's been a pleasure talking to you, looking at your homelab, you know... Going through that stopping and starting phase. That was very, very fun. And I'm very much looking forward to the next one because I haven't shared mine yet.

I mean, I really look forward to seeing your homelab. And we have to do a round three. We will. Yeah. Yeah. For sure. Thank you. I'll see you next time. Thank you, Gerhard. Bye-bye.