How Do You Build a Hybrid Home Lab? - C2C037

I'm joined by these two chuckleheads Tim McConaughey at JuanGolbez on Twitter he's only on Twitter and Chris Miles at BGP Main , who is also only on Twitter .

So we're all long time network nerds and a big fan of all of you all in this show , so really happy to be here .

We've had the most guests on this , on this podcast from Kentucky , which is close to close to my heart .

A lot of people are kind of stretching their home labs between their house and having some kind of presence in one of the public clouds or multiple public clouds , and it can be a mess , right to get everything kind of connected and figure out exactly how you want to have that portability and everything between here .

And Matt is actually working on a project that he's been doing for a while , so we wanted him to come on and kind of talk about some of the struggles that he's had , some of the decisions he's made , some of the decisions he hasn't made , and it's still , as Matt will tell you , it's still very much a work in progress , right , but you got it pretty fleshed

out so far . Do you want to just real quick , I guess , talk to us about what you're building and what it's for ?

You know , and I never really went full in , I always just had , you know , a few computers sitting around that would run things on . But working in data centers with real labs and you know , almost every job I'm in there's a lot of space in the data center , there's place to play and build things .

And so I was like I'm going to run a small infrastructure , connect it up with the VPN and be able to kind of burst to somewhere else if it's just work , or then maybe the cloud . Again , this was in 2018 , so I hadn't really fleshed that out , but even back then I kind of had this idea .

It was something that I wanted to be able to have a small footprint locally . That was somewhat self-contained , but as I needed to do more , I had a way to burst out to those resources , just like people talk about Cloudburst today , but on a smaller scale and just using simple technology .

So that idea had been around for a while and I've been wanting to test some things out . My one trusty server , I think the motherboard died . It finally kicked the bucket and I sat around for a few months not doing anything and thinking , oh , I'll just fix this whenever I didn't know if the power supply died or what .

And eventually I diagnosed it and I have a gaming PC as well , and so I just sort of run some containers on there and quickly got to the point where I was like the right way to do this , I need something that's reliable . Other people use this computer .

So I went online and found a couple inexpensive components that were on sale at the time , just something that could essentially be a few Linux servers that could essentially just run containers .

I don't need anything fancy , not running VMs , just Docker containers or Podman containers running on Linux hosts with some very basic configuration and just keeping things essentially like stateless containers , store all the config somewhere that is backed up and replicated . And that was just the general idea that I had .

I just , you see , this idea repeated a lot , you know through Kubernetes , and you know people build giant solutions using this model and I was like I want to build , kind of build a home lab to do this , but I want to start by automating it from the very beginning .

Right , build out everything with Ansible or some scripts or whatever , because I had already rebuilt this lab many times in the past , right over the years , and I did it different every time . And there's other frameworks out there .

I'd even used some of these other popular home lab frameworks , but they're definitely built for the masses and I wanted something that was a little bit geekier , a little bit more flexible than what was out there , and I also wanted to just learn some new things . So there's a couple of technologies out there .

I'd use Docker forever , but I wanted to learn Podman and I wanted to learn about doing it in a more secure way , running containers rootless instead of as a root user , things like that . I just wanted to explore some of these ideas as well .

And then , obviously , being able to , just like I said before , have a small infrastructure in my home and then deploy essentially to the cloud , free tiers .

That's where I'm starting out , just doing free tier deployments and connecting it all up with Tailscale for networking , all automated , everything comes up and connects once it spins up in the cloud , and then that's kind of where I'm at today , basically with built up a lot of automation , either Ansible and Terraform .

A few other things I've only really experimented with today GCP and Oracle Cloud . The other all have their own free tiers that I'll kind of move into soon . But that was enough to kind of get started and the idea is to open sources at least and share what I've built so far so other people can tinker with it and try it .

So you're talking like just going to eBay or something and just buying a workstation or something , not like a huge rack server or anything right , just something not like a Nook , or maybe a Nook if they still sell . I don't know if they still sell Nooks , actually they had discontinued them .

So three is the smallest odd number you can start with . That can be threes . That could be three little physical boxes . Actually , what I have running today two physical boxes and a VM . The third physical box is on the way . I just kind of spread out the cost over time so it's flexible enough that you can kind of do it however you want .

When I was prototyping this and just building the initial automation right Like building with Packer to create a VM , like I just got that process down and ran it with very little resources attached to it because I wasn't doing anything else , so I could just kind of run all those locally .

But then , as I started to layer on more , things grew beyond the bounds of the little machine that I was running these VMs on and I kind of did a V to P basically on those little servers I built . I actually just stood up the physical servers and ran all the Ansible playbooks that I'd already written .

They ran on my VMs , they set them all up and once the physical servers came in , I just ran it all again . That was the whole point of doing all this tooling , mainly with Ansible , completely modular A couple of the things there . But as soon as I get something in that I need to set up or I need to decommission something , it's very easy to just like .

I got to stand it up and get an IP and then the automation runs and it configures everything the way I need it to .

That basically what I'm doing is I'm writing a Terraform to deploy , based on the exact specs that are in the free tier of these clouds , and so GCP , it's like one VM this size , blah , blah , blah , blah , blah , and then there's a bunch of other things that can be run in the free tier .

So it's not hard to write that Terraform and deploy it and then you just have to script the install of the tail scale and the other stuff once it comes up .

Oci has a very generous free tier and they run a few VMs and they let you run a few VMs and they even they actually , um , when you can spin one up , because their free tier is , uh , very popular now because they they do give away some resources , they have , um , um , some arm based VMs that you can run , uh , as well .

So you can run Intel and arm VMs and OCI and I thought that would be kind of cool to tinker with . And so the only reason I hadn't moved into the other ones yet is it is a time investment to write Terraform to deploy everything .

Amazon and Azure's free tiers are a little bit less useful not useless , but just not as pertinent to the stuff I was doing to start out with . But I would like to have something running in all of them , right , because there's . Even if there's not much there , there's . You can run some monitoring .

You could run , you know , little projects here and there and also the way that I have it set up , you're going to have routed connectivity into all these clouds .

So if you don't , if you need to move past the free tier and you want to pay for a service , you know , just maybe not pay a lot , but you could go and then consume those various different cloud services , whether that be some sort of new AI thing or just you know , the standard cloud services that we all know and love .

But instead of , you know , going through over a public IP , we've built out this network with Tailscale and so everything's private and it's also very easy to join that Tailnet from like a client PC . You just install Tailscale there as well and then you're connected to your hybrid home lab .

There's Dockerized versions of all kinds of different network OSs . So just in terms of network engineering , having a platform that you can just spin up some containers on is cool . It's useful . You can kind of just do what you would normally be doing for work .

What I'm using it for is to do a lot of learning . Right , it does have . I use it for some professional things and starting with some home automation . So I have like home assistant running in my lab , which is not really related to technology , but I wanted to learn it and it's easy to spin up there .

So , but most of the things I'm running are the things that you hear talked about in IT all the time . Right , I'm running a few different data services , like Redis and Postgres are there , because everything uses one of those two things .

I'm running some network monitoring like LibreNMS and Prometheus , and some logging solutions , and so it's sort of that basic IT infrastructure stack . I'm using one password for all my secrets management . That's not really something that's a . SaaS service . Right , I chose that just because I already had it , so you could use something else .

You could use Vault , but I wanted this reliable container platform to run whatever I needed . But I've been focusing on learning a lot of the different AI and LLM tools out there . I mentioned I have a gaming PC with a GPU in it so I can run some small models there .

And all of these new AI tools whether it's just a chat interface or it's a RAG solution or anything along those lines it's all released in a container format .

So I wanted to be able to spin those up , because it's the wild west right now and there's new things coming out all the time , so I want to be able to try stuff out there the quickest way right there's some cool like diagramming things and stuff you can self-host , right .

So I mean , as engineers , we're having to create diagrams a lot and we've used all these different tools and there's some great self-hosted options there as well . Um , rss readers you can self-host . I'm paying for an rss reader now I'm gonna self-host one . I'm paying for an RSS reader now I'm going to self-host one .

There's even a tool somebody wrote that will take your RSS feeds and put them in an email and email them to you , which I pay a service to do that for me today . You know there's a huge community out there that's building all kinds of things you can self-host . So the imagination is almost , you know the limit .

You can write your own code and package it in a container and run it too . I mean . So it's just real flexible . And you know , like I said , just a lot of the typical IT use cases learning , ai and LLMs , and then just some hobbyist things , and there's the sky's the limit , right ? I mean there's so much more you could do .

So let's go through some of the like the IAC tooling I know you mentioned , like Terraform Enhanceable , but are you writing your own scripts , Like , were there any challenges with getting this set up ? Basically , what are , what are the tools you're using for IAC and I guess what are they used for in each part of your setup ?

And so they have also a container scheduler , nomad , and they have a service delivery tool called Console . Those two things work very closely together , right ? I mean those would sort of be the replacement for Kubernetes in this solution .

So instead of using Kubernetes for container scheduling and service delivery , that's Nomad and Console , and they do some other things as well . But really the bulk of what I've written and the bulk of the automation is Ansible and Terraform . Ansible I've used for a long time .

It's really well supported to do just about anything right and its strength is things that you would do on the CLI . Typically , from what I've found , getting on a Linux server and doing some configuration , ansible can handle that really really well . And so that builds out and installs all the dependencies that in my lab .

It installs Node-Added , installs console , it configures those , it installs Tailscale . So basically Ansible stands up everything after the base OS is installed . Once that's set up , terraform is used for more of the API-driven work deploying to the different cloud providers . We're more of the API driven work deploying to the different cloud providers .

It can you know , since we're using a lot of things in the HashiCorp ecosystem right . Terraform understands Nomad and console really well . It can easily deploy jobs to Nomad . It knows exactly what to do . It's just a few lines . So those are the main components that I'm using there . I mean , there's some Python in there , there's some Bash .

As I was building VMs , I wrote some PowerShell and I took some examples . I took some inspiration from people who had done this work . It was a little bit dated and needed to be updated .

So I got to work , sort of taking a few different examples and getting them working in that situation , and then I also brought in what I refer to , as you know , my super genius via SaaS . I subscribe to Cloud AI and you know OpenAI would work as well .

I just started using Cloud and I found it worked really well , so I subscribed there and that's basically my assistant . Like whenever I get stuck or I don't understand what to do or there's a weird message or I need to do something really strange and answerable . Then I go ask the super genius and it usually gets it right and not always .

And so sometimes there's some back and forth and some troubleshooting . But boy is it good at troubleshooting and sort of what we might call test driven development . I guess you know you can kind of say I want to do exactly this and it'll , it'll start working at that , and you get some results back and you say these aren't exactly what I'm looking for .

I can take that and fix it . So I have an assistant that helps me whenever I get stuck , and that's why I think I was able to get this actually somewhere close to a finish line . I think if I was doing this completely on my own I would have gotten stuck and kind of gotten demoralized , or it would have taken me months and months longer .

I mean getting started . The initial work that I did in the first couple of weeks , I know on my own would have taken me a few months to do . So it's been an awesome enabler .

You want to open source this and have other people how much of a finished product are you thinking of delivering other people like how , how much of a finished product are you thinking of delivering and how much of it is ? You know something that people are going to download and have to to diy it . Uh , and and there's value in that too right , in diying it .

You've obviously learned a ton doing it . So , kind of , what's your vision there ?

I've also put a lot of time into just sort of building out these job specifications that are a lot like a Docker Compose file , but some of them are much more complicated because Nomad does a lot of stuff , and so if you set up all your own infrastructure on your own and just wanted to run Nomad , you'd be able to take the things that I wrote to , be able

to deploy these various tools and spin them up there too , so you could take the whole thing and deploy it , and hopefully , once it's sort of maybe at a version one , then it would be very easy to do . It'd just be like fill in all the variables that you need , deploy your servers , run your Ansible playbooks , and then the infrastructure would just come up .

It's not quite there yet , there's just a little bit .

It's still in a science project phase , but the more I've used it , the more I've improved it and I've definitely broken the whole damn thing a few times and just completely redeployed it , and every time I do that I fix something and I'm getting closer and closer and closer to it , to being something that's actually reliable and works all the time .

You know , there are occasionally just times where a service stops working and I just restart it . I don't mean I could go investigate it . There's always things that could go wrong . There's , there's always funky little things in there , and a lot of times I'm using software that was just released the night before , sometimes Right when I'm downloading a container .

So there's weird bugs and stuff that come up . And I think if you don't have you , if you're not real comfortable digging in and troubleshooting some of that stuff , then this is going to be a heavy lift .

But if you've been in technology for a little while and you just want a container platform and there are many of them , right , so this one is very opinionated , you've got your choices . But this I'm trying to thread the needle between simplicity and capabilities . And Nomad and console are pretty easy to deploy .

They're like one go binary versus deploying Kubernetes is quite a project on its own , and so not to say that I haven't done some very complex things once I stood those up . But I wanted to start simple and sort of layer complexity on top of there , so you could take this and start very simply and do some simple things .

But , honestly , I'm here and talking to you all because I'm also trying to figure out what to do with this ultimately as well . I don't , honestly , totally know .

I've spent a lot of time working on it to the point where I'm like , well , I should show this to a few people and see if anybody else is interested and would certainly eventually love to have some people collaborate on it as well , or just try it out , kick the tires and tell me what .

Building right is going to be for people that want to tinker and want to self-host stuff , and I think that's very cool and I like the extent that you've taken it , and I don't even think we've said this to this point , but you've even given it a name , right , you're calling it Octant . Is that the name that you've landed on ?

One thing I want to know about is obviously the tooling that you chose for this particular project , this particular home lab . I'm sure cost came into a lot of this .

I see you're using a lot of tools where you can at least use a free element to orchestrate a lot of this piece , but what went into some of that decision making , if I think about the audience who's not only listening to this but probably also on this call ?

I've been a big zero tier fan for a very long time and I know you had some internal notes about why you went with tail scale versus zero tier . So what kind of went into the decision making on why to choose one tool over the other , et cetera ?

Let's take Podman as an example . Everybody knows Docker quite well Most people do . I'd heard about Podman , I'd used it a little bit . I wanted to learn it and I knew that was a drop-in basically . So that didn't seem like a difficult decision . I learned something new . That didn't seem like a difficult decision . I learned something new .

It's about the same as the other tool . If I don't like it , no big deal . But I've actually turned out to be quite a big fan of that in particular for a few different reasons . But tools like Terraform and Ansible there's muscle memory there for me . So that just came down to a matter of taste and I didn't want to .

You know there's already a lot of tools in play here . We've talked about a bunch , and so you run the risk of over tooling sometimes , right ?

So I wanted to make some clear decisions , but not still keep it somewhat limited to this specific scope and not overdo it , and this was sort of the minimum I could come up with to do all the things that I wanted to do . You know I'd made some odd decisions . I'm using a ton of Hashi products , but not Vault . I pay for one password .

It's got some cool DevOps capabilities . It can basically function in place of Vault not completely , but it will basically do all the things I need , and so I'd also struggled to make Vault work in the past and I was like this works , basically do all the things .

So it'd be hard to take out Tailscale , It'd be hard to refactor all the Ansible and things like that . But there are some like one password could be swapped out If you . You know there are a few different ways to do this . If you wanted to , you could take out Tailscale and try to do zero tier .

But I ended up and we should talk about some of the networking stuff that I learned working with Tailscale . But it does some amazing things and it's very easy to use and it's using WireGuard in the covers , which is modern and fast and can run in user space . Even so , if you've got a locked down laptop , you're not allowed to install a VPN client .

You can run Tailscale in user space and , nobody's the wiser , you can connect the VPN . Are there questions that you have about specific tools ? I mean , I kind of talked about some of them , but did you have any questions about specific choices that I made there .

So , what issues did you run into and how does Tailscale kind of help you solve some of those ?

I'm using Terraform provider to interact with Tailscale to do some things right To provision machines . So there's a lot of great functionality there that I'm not sure exists in zero tier . I haven't looked recently here . I haven't looked recently . I know there's an API but I don't know if there's a Terraform provider and stuff .

So just standing up the basic Tailnet and connecting machines , it's very easy . But I quickly was like well , I don't want to install Tailscale on everything in my home to be able to get to these resources that might be deployed somewhere else and so you can advertise and you can route traffic across Tailscale .

I think the first thing that I noticed that really kind of sent me for a loop . And this is a combination of a platform difference , right , so different behavior . On Windows and Linux , when you accept a routing advertisement , it takes presidents over connected routes because the way that Tailscale does .

They run multiple routing tables , basically , and they do some policy forwarding on Linux . And so as I was doing this initial troubleshooting , I was like , why can't I get to my connected networks ? These should always work right . I mean , this is like CCNA day one connected routes , everything else goes from there .

Well , not Tailscale On Linux when you accept a routing advertisement . So the details are sort of in the answerable I wrote . But I just had to be very specific about which device in my home network was going to act as a router , which other ones would accept routing advertisements or not . And so once I got that figured out , it wasn't that hard .

But it was like what ? Like this , it didn't make sense , right , I've also , they've made over a couple . There's been at least once where there's a change made that broke things , and they're going to go in and and and re reconfigure the , the options because they I think they patched a vulnerability and they changed away some of the behavior , and that happens .

But it kind of felt like a little bit of a rug pull . Everything was working and then suddenly it wasn't , and so I had to go fix some things there . But beyond that , I'm trying to think what else ? Oh , the other thing that I had to figure , and I'm just basically doing static routing .

I have just a Ubiquiti router that runs Viata flavor and so I can just configure some static routes there that point to my tailscale router for the cloud networks , whatever they might be . Eventually there's BGP speakers in all these places . I think I'd eventually like to stand up BGP so that routing is .

You know , it's all dynamic , but I just have been doing static routing so far . But so the other piece , the other big piece I had to figure out , was essentially had to enable TCP MSS clamping as things route onto KSCL because there's a VPN there , right . Things route onto a scale because there's a VPN there , right you , it's any .

I think I set it at like 1380 or that was a recommendation I found , and once I got that in place , everything works . Everything on my home network can get to anything in the cloud that's running that's connected to tail scale automatically . It just works . We can monitor everything . So that felt like a big win , honestly .

Like once I finally got that all working , I was like , okay , this is cool and it's free . And like that's when the light really came on that I was like man , you know this is all free , but , like , if I ever want to spin up services anywhere and use them for a slow while it's so easy to do you can consume them right from home and then you know .

Honestly , like I'm just interested to see , I want to know what other ideas people have for this right .

So you could certainly , let's say , if I wanted to give you all access to my home lab , right , I could invite you to my tail net and I could just configure the ACL there , that's , so you could only get to port 80 and 443 on my ingress and not get to anything else , and then that would generally work . And I'm probably going to start doing that .

So with a few friends and hosting things locally . But otherwise haven't focused a ton on security internally , just because nothing's exposed . But there's certainly that's a place where things could get better Internally , just because nothing's exposed , but there's certainly that's a place where things could get better . Tailscale is really capable there too .

I just haven't really dug in too far beyond just the basic sort of ACL and firewall and capabilities that they have .

What can you tell people about setting this up and some of the challenges and difficulties you've faced along the ways ?

The minute you screw something up , you can just rewind it in time and it can even apply as like I'm writing documents , I'm writing them in Markdown and I'm putting them in Git and I can see how they evolve over time and stuff . So that's indispensable

. I said I you know I leverage the AI quite a bit . Like I didn't just use it to write code . I would sit down and be like what do you think about this idea ? Break it down for me . What are the pros , what are the cons ? Like what haven't I thought of here ? Okay , that looks good . Generate the pseudo code for that .

What would that kind of look like ? And I just kind of used it as like a really smart friend to bounce ideas off of , and I think that allowed me to go from like an idea to a working solution in like an hour , a lot of the time , versus for me it would take hours or days or weeks to come up with some of the stuff that it , that it spit out .

So use that force multiplier . You know whether it's it's free or or you're paying for something , like Claude , like if you're writing code or you're doing technical work .

Unless you yourself are a genius and you have the whole internet's knowledge in your brain , you can go Google errors or you can just paste them into an LLM along with the code and say why did this error happen ? And nine times out of 10 , it's like you screwed this up . Right here you fix this . Here's the fixed code .

So those two things allowed me to get this to where . It is much faster than I would have been able to do it typically . I mean , I have a degree in computer science . I don't write a lot of code . I haven't since I was in college really . I've been in technology and infrastructure and I understand it and I can read it and I've written plenty of code .

But actually sitting down and writing a ton of code or a ton of Ansible or a ton of Terraform is hard for anybody . And if you don't do it all day , every day , it's just slow going . And when you have the super genius that can spit it out for you and it's , I find , somewhere between 50 and 100% correct .

It varies , right , but it either gets me about halfway there or all of the way there . Otherwise it's just been . It's a fun opportunity to learn something new , and so that kind of motivated me . I don't know why I chose now to do this . I think it was just a combination of all those things I mentioned .

Like HomeLab died , I wanted to rebuild it , I wanted to learn these things . But going through this process , learning a bunch of cool stuff , solving some hard problems that's the fun part about these jobs .

I like solving hard problems , so like pulling apart Tailscale and figuring out how it worked and getting it to work , and like route between home and cloud Like . For me that's that's satisfying work , it's fun to do , and when it works it's like well , that's cool . On to the next thing .

And you know , all this is captured now in a GitHub repo and so at least worst case scenario , it's like a time capsule of things Matt learned over those last few months that somebody maybe can go back and look at later and use . That's all I can think of right now , honestly .

Here's exercise three we're going to build the Ansible scripts . It's like Kubernetes the hard way . Yeah , exactly that's what I'm hearing , honestly , and I don't know if you've considered that it also would make it infinitely more consumable to the people , I think , to have a of a map to go .

So I don't know if that's something you're interested in , but it definitely , I think , and it would teach a lot of people a lot of things along the way , the same way that you learned a lot of things along the way . So that's my feedback to you .

So I guess , from your perspective , in what you've built so far , you know , in which I don't think we've even touched on this yet , but we will put , at least whenever this comes out , we'll put at least everything that Matt has up to date into the show notes where you can find more , where you can look for when this will be posted out there .

But I guess , staying on the topic of clouds , obviously you've built something that is extensible , not only from on-prem into the cloud , I guess , and in other words , you could also just start this directly in the cloud without ever having , you know , on-prem presence , and you could extend it and on-prem in the other direction , right ?

So what do you think are the key benefits to doing this ? From a home lab perspective , for someone that wants to build this , like , what is the overall benefit that you've seen , having this extensible home lab that can pretty much go anywhere ?

But then , you know , I've thought about ways to use Raspberry Pis to do some certain things and you can scale that in a bunch of different directions , right .

So I mean , we've got this automation and essentially , as long as there's something Linux that you can run Ansible playbooks against , or there's an API to consume , right , there's something in this tool set that will allow you to , like , build , you know , a place to run your workloads .

I mean , when it comes down to it , like all of the things that all of us are doing , in the end the goal is to run a workload somewhere , right , get something done , and so I have a flexible platform that allows me to run workloads anywhere , almost right , just .

And I love the idea of you know , just , I deploy something with Terraform into the cloud and then , as soon as it comes up , it's just connected . I don't like , I didn't , I wouldn't even have to log into the cloud console after the , you know , after it's built , and so I like flexibility there , I like being in .

With Terraform , it's easy to bring up and bring things down as needed , like if I'm not doing anything , why leave it running If it's very easy to spin back up and rebuild , like so there's , you can mix and match to do whatever you want to do . And that's not for everybody . Some people just want this is you know ? Here's the recipe , here's your cookbook .

Go do it . This is a little bit more . Choose your own adventure as well , like you can kind of pick and choose what you want to do . So that's , that's , that's what excites me All right , awesome , yeah .

I want to just see what people think about this . But I'm also like I got to fix this one other thing . I got to deploy this one other service . You know there's always something that's like keeping me from coming and doing some of the toil and documentation and stuff .

But getting on here and seeing you all and catching up and talking about this kind of got my butt in gear to do that and I would love to come back and do a walkthrough and demo . I think that would be awesome and then we could look at each of the components . Storage is one of those things that could be anything .

I'm using Ceph but there's other options , but we can look at that and nomad and console have a nice easy to use ui and and then we can look at some of the other tools as well . I've built some cool network maps monitoring everything you know , all the network nerd stuff that we do . So awesome .

Repost it everywhere on your social media TikTok , as Chris always likes to remind me that we have . Just share it around and give us a good rating and likes everywhere . So thanks for tuning in and we will see you guys next time . Hi everyone , it's Alex and this has been the cables to clouds podcast . Thanks for tuning in today .

If you enjoyed our show , please subscribe to us and your favorite pod catcher , as well as subscribe and turn on notifications for our YouTube channel to be notified of all of our new episodes . Follow us on socials at Cables to Clouds . You can also visit our website for all of the show notes at CablesToCloudscom .

Thanks again for listening and see you next time .