325: renderDEEZ128

Okay. I just did I did some Edge of the seat beep work there, Brad. Yeah. What does that mean? Well, I started the recording at like six seconds to go. I usually try to start it at about two seconds to go so that you don't have to like, you know, that I'm the short one instead of the long one. Same. And uh I realized I had to delete, I had to stop it, delete the file that was there, the line that was there, and then restart it.

with like two seconds to go and I got it with maybe maybe half a second on the first beep. The hidden the hidden beep etiquette that emerges when you podcast remotely all the time. We're talking about the little um I think we we refer to it internally as beep thirty.

Yeah. Which uh is the tool that was built for us by Thristheart. Is that how you say his username? That's right, Thrist Heart, yeah. On on the Discord, which is the thing we use to sync our recordings together. Anyway, yes, I also I always hit the record button like right before the beat starts so that It's right at the beginning and especially for if I'm not editing the even even when I am ed editing the podcast it's useful, but especially when I'm not

It's right there at the front. You don't have to look for it. No, no. You you don't want to have to scroll a minute in and then find it. That's no good. But um but that said I know exactly what the waveform of those beeps look like. It's like burned into my memory at this point. Oh yeah. It's the most important waveform I ever see. I was gonna I was gonna ask you.

I don't actually remember what I was gonna ask you now'cause we started talking about the beefs and I was like, There's a mosquito in my office that I've got to deal with. The the the the prospective cold open topics discussed right before we started this were there's a mosquito in your office and you're down to Two micro USB devices. Oh. Right. The micro USB devices. So

I have a perfectly good USB like PD two Oh no. I see you holding it up right now. Pat pack. I see where this is going. This is I yeah. I already am experiencing the pain of whether this is worth it or not. Yeah. It's it's great. It will charge like the entirety of anybody's phone in the house almost. I think the I think Gina's big iPhone Max has a bigger battery than this, but like my daughter's mini and my older iPhone Mac. Both will will fully charge off of this thing.

It's a micro USB charger. And I have there's literally on my charge station over here to the right, there's one yellow cord remaining that's for like the three micro USB charging devices. I have one thing that's Let's say Uh legal drug paraphernalia. Uh huh. That's micro USB. It's a it's like a it's like a flower vape. It's nothing it's nothing weird. I believe um and then I have

I that I don't really use all that often, so it's not really an issue. And then I have this battery backup and I have one other battery backup, but the other battery backup. has both USB micro C and a microcharger. So it's fine. It gets to live. It can charge off either port. It has two Two ports for charging and two ports for draining. Oh, I thought typically the batteries only had one port that would charge. That's cool.

Yeah, so it's it's from a weird time. It's from like right as the as the changeover was starting, but before like iOS had switched over, IOS devices had switched over. Um so what's the quandary? Is it whether to keep the micro only bat? I just saw the mosquito fly right across your face. It's making me crazy for fucked.

Yeah. You're you're boned, I'm sorry to tell you. Everything's collapsing here, man. It's bad. Um I d what what is is is the question whether to get rid of the micro only battery or Yeah. The question is is it is it like I generally on stuff like batteries, I try to keep'em until either they get puffy and stop working or they stop working, right?'Cause like

There's no downside to having a battery that's like eighty percent capacity or whatever. That that is well, I would say the same thing is true of a battery that is limited to micro USB. Let's say they are batteries are too useful as a as a backup. I mean I don't know about you, we don't have that many power outages around here, although certainly more than we used to, but like it's just too useful to have extra batteries around.

And even like even if you never use that one hands on, like even if that one is just a charge it once every six months and leave it in the drawer for emergencies. I st I still feel like that's not a good reason to get rid of something that's otherwise quite useful. Well, so the problem with this one is because it's the smallest, kind of crappiest one that's in in service still. Mm-hmm.

It's the one that when the kiddo is like when I'm like, Hey, did you charge your phone last night? She's like, Oh yeah, I got it. I'm good. I'm at eighteen percent and we're getting ready to go out of the house for three day three hours. I'm like, No, you need to take a battery back up and plug it into your phone in your bag and she's like, Oh, I don't want to do that. Has she ever actually taken a battery with her?

Oh yeah. Yeah. Like when she goes to when she goes to class or something and sh and she has like ten percent battery, it's always dead. Can I borrow battery? And I don't wanna give her like the hundred dollar big chonky one that'll do laptops and is good for like four or five tr phone charges. A because it's heavy and big and B because it was kind of expensive and it's like

That's like our disaster preparedness one, right? Like that's the one that I can take out. Like if we have a bad problem, the power's out for a week again, like it was in 2020 when the fires happened. Um, I can take that one out and I can charge it off the car battery and then bring it back in to top up everybody's phones and stuff. So I feel bad getting rid of it. Yeah. I feel like it's wasteful. Sure. I kinda c I kinda feel well, you could give it away to somebody or

I'm not gonna give somebody a USB micro USB. I mean all all I mean is if you don't wanna waste it, if you don't want to literally junk it, you could find somebody who would use it if you wanna feel like it's still being put to use. Maybe I should use it for like a project. Maybe I should get something that's like I saw I saw one of those um kind of ambient display screens the other day. And one of the options for that was to hang a USB battery in it and and like

I bet that this battery would run that thing for like four months. Oh yeah. As little power as it draws. My Kendall, I have not I I read my Kindle every night sometimes. Only I usually only make it ten, fifteen minutes before I fall asleep, but Yeah.

I haven't charged my Kindle in like six months or something and I just noticed it was it was down to like forty percent the other day, but still that's like six months of Kindle usage, so yes. And that's with a backlight, so I think like a non-backlight e ink screen. Would probably go forever. Do you put your Kindle on um airplane mode? Sorry, yes, I should I should point out it's it's always an airplane mode unless I need to sync to it'cause the battery lasts so much longer.

Gotcha. And also this is a topic for another time, but Amazon has gotten incredibly aggressive about pushing auto updates to that thing.

I I almost always just read on my phone or my iPad now. Really? Um, but I also like I tend to be like I find myself in places where I have like ten minutes where I'm like waiting. It's not The the classic example is oh, I have to have the oil changed, I'm sitting there waiting for the oil change to be done. But like I that's not really a problem with the electric car anymore.

Um but but yeah, like I like if I'm at a doctor's office or something, I always sync'cause I always want to have my loc my latest location so I can just pick it up on the phone. Fair sense. Yeah, I get it. The magic for me is Also, I'm doing almost exclusively Libby these days. I don't think I've bought a book from Amazon in a year. It's actually I know when the last time I bought a book from Amazon was. It was when they did the the changeover

On the D R M stuff. The D R M changeover because like I also have not bought it. I I I successfully ran whatever that little JavaScript applet was pulled all my stuff off, which I have archived now, and I probably will never buy another book from them because of that. Sh uh also, shout out to Libby for cookbooks. I didn't realize this. But Like the San Francisco Public Li Library and San Mateo County Libraries both have really, really robust E cookbooks collections.

So like when I was looking at the tartine bakery, the fourth edition of it,'cause I have the second edition and the recipe I wanted isn't in that one. It's only in the fourth edition. It's like a$50 cookbook. And I was like, man, I'm gonna have to buy another, I don't wanna have another$50 cookbook on my shelves. And then I was like, what about Libby? I searched Libby, it's there. Nobody had it checked out. They had like eight copies of it.

And I got it. I copied and pasted the recipe out and I put it in my in my paprika and then I just returned the book immediately. It was great. Wait, hang on, is that a Libby interface to a local, like physical library system or is that just some like cloud libby fli cloud library that that does not actually exist.

So li you could do you can do both, right? Like there's virtual libraries that you can connect to with Libby. You can also connect to your local interfa your local system. Like I when I check out an ebook from the San Francisco Public Library, I add it to my Kindle with Libby.

Hmm. Have you never done this? I I assumed as a library aficionado you would have been on this train a long time ago. Like I said, I use books largely to fall asleep at night. I wish I had time to read more, but I've kind of have been reading the same Three books off and on. They're like short story collections and one very long history book. I mean like a seven hundred page history book.

For a very long time. So I just have not obtained any new books. You're still looking around at that mosquito. Fucking mosquito. Anyway, I uh I have not l really used the SF library since they started transitioning off of they were used to be on like overdrive. Oh. And Louie's dope. Yeah. I should try it. I should get back into that.

Uh it it took like five minutes to connect my card to the to the system. So just to close the loop on this whole cold open'cause we've got to start the show. Uh and this has been like five cold opens in one. I the real reason I fell off of messing with the Kindle is the last time I tried to jailbreak it.

Oh. And to go back to the auto updates thing, maybe we should do an update on a future episode about this. I washed out of the jailbreaking my Kindle process because it auto updated in the middle of it. Oh my god. To a version that could no longer be jailbroken. Oh that stinks. And that kind of soured me on the whole s experience so much that uh

Not using the Kindle as much for the last little bit. So uh you saying you uh tried to jailbreak your Kindle but you ended up putting your Kindle in jail? Mm, yeah, hurts it's more like Kindle jail broke me. I don't know. What podcast?

Welcome to Brad Will made a tech pod. I'm Will. I'm Brad. Hi, Brad. Hello. Hi. Um, this week. Mm-hmm. We're doing something a little bit different. We haven't done this. We haven't talked about this in kind of a long time. We also kind of talk about it a little bit all the time. Yes. You're not wrong, but this is this honestly this episode is the genesis of this episode is by user request.

That's true. Uh we're gonna we're gonna do kind of a I guess this is this our beginning of twenty twenty six kind of home lab update. I don't I have to confess, I don't love the term home lab. I mean I don't hate it. Like it it it communicates the concept well. Actually what I really don't like is having spent a lot of time on the TrueNAS forums in the past where people

post gigantic sigs of their like three massive servers with 96 hard drives each. Look, SIGs are always bad. And like the term the term just evokes like um What's the concept I'm trying to say here? It's not it's not a it's not a um form of or function thing, but it's kind of analogous to that of like a Well what is what is the term for like like a c amassing way more capability than you actually need? Like way more. Overkill? Sure.

I mean th there's a there's a there's an arms race in the like if you go to R slash home lab or one of those places where where it's like Somebody somebody posts their, you know, thirty-six inch tall rack, you know, a twenty-four U rack. And then somebody else is like, hey, check this out. I my work was throwing this out and I got a forty-eight inch, forty-eight U

twelve foot tall. It's gonna be fifteen hundred pounds when I put all the computers in it, rack. And yeah, my my one petabyte storage server. You know what it is? It's hot riding. That's all it is. That's all it is. It is just tricking something out way beyond

any any practicality or necessity. But but I mean I think that there's another so there's a there's multiple aspects to the whole home lab thing, right? Because the other thing is people use them as learning places and For example, back in the old days at Maximum PC, we never our our our business people, our ad salespeople are always like, hey.

Could you do the thing that PC World does and have like a big info world type can you do some server testing and stuff like that? And we're like, no, we don't have the we're not set up to do that. It's too expensive. Like we we don't have the capacity to build the 300 machines we need to test a a web server effectively, right? Or or whatever it was. And um the home lab community has kind of used

I don't know. They're not special purpose, but like g used computers and stuff like that and and and come up with some best practices that let you do that kind of stuff and build that kind of expertise and build like, hey, I'm I'm building a big giant infrast web infrastructure or server infrastructure.

So that you have a place to practice that and test that and fool around with that totally as a personal person, which is cool. Totally. I mean, A, okay, A, to be clear, people can do whatever they want with their money and time. B if they're actually you know, making practical use of the thing they have built even better. Yeah. But I I just personally don't want to feel like I

And building all this stuff out just to sit there and admire it. And like I kinda try to make sure I'm actually making use of what I'm doing. Like for example, I kind of overbought hard drives. Yeah. The year before last at Black Friday. I expanded my storage pool. Yeah. Uh to to the point that I like it's still sitting there half empty because I at some point I finally realized, you know, I may have bought too many hard drives at this point.

But so so to me it's like it's like the people who are the hot rotters, the people who just want to have like the biggest, baddest machine set up, right? And then there's the piracy aficionados. They're running a whole stack of R's. See, so those people with petabyte storage volumes are actually using it for something. Yeah, like the ethics of that something aside.

Yeah, it's like it's like Steve, Steve from uh from North Carolina wants to roll his own Netflix. Got it. I understand that. It's not for me, but I get it. And then there's the people that wanna learn. There's like the people that want that are like like are you building stuff so they can get certifications or advance in their job or whatever, which is great. Kinda I I guess that's kinda where I fall these days. I mean, I'm not trying to get certifications, but like I was

going through some networking stuff in the network channel the other day and people started talking about having done that stuff for their C C N A. Mm-hmm. Uh it was a bunch of IP V six stuff that I was poking around at and it was just like, okay, I guess I guess that's kind of what I use this stuff for is I just like learning. Well, and then there's also people who want to kind of get off of the public

like the the corporate clouds, right? Okay, that's all people who want to divide of Google. I think I fall in that category a little bit. Or at least have a b have a have an alternative to in case you decide you want to pull the plug entirely. Yep. Um and uh and so okay, so there's there's motivations. That's what people are doing. Yeah, so so so real quick, the this this all started from James on the Discord

uh adding me a question which was, hey, can you run through your Linux NAS, like your Linux server software stack? Because I run TrueNAS. I'm speaking as James here. This is James saying I run true NAS, but I'm like kind of over it. I'm kind of thinking about moving to something else.

And I'm enticed by this idea of just running a bare operating system and doing everything yourself, but I'm also kind of afraid or intimidated by it. Are you gonna get it? No. He's leaning back. You gotta stop asking me about it. It's an audio podcast. I know, but this is like I know it's high drama. This is action cast. Yes, high drama is the term It loves the lights. I'm gonna go and tell you it loves my lights. It's a flying bug. Yeah. Of course it does.

Uh I I need a bug assault. I know I know formation as if you were about to get it, and I just couldn't help myself. All right. Anyway. James was basically asking, Hey, can you step through like the software stack and how you've configured everything on your Linux box to be a NAS slash server uh without the aid of an appliance style operating system? And that was kind of the genesis of this episode, but we're gonna

kind of expand it out a little bit and just sort of do a quick home lab overview of everything? Yeah. Well, I I thought it was interesting to kind of do a reset because w like pe longtime listeners will know that in a lot of ways this podcast started with us talking about home labs. We started talking about true NAS or FreeNAS back then. Sure. Um Yes, cause this podcast started I built my TrueNAS machine in April of twenty eighteen and we started this in September of nineteen.

Yeah, the the origin for a lot of this was you and I were spending a lot of time on Discord talking to each other about server stuff. And finally we were like I was like, Hey, do you think CBS would let you do a podcast? And you were like, I don't know, let me ask. And then

You asked and then it took them a year to decide and then we did a podcast. Yeah. Yeah. And and and like you and you and Vinny were the the people I knew running FreeNAS. Like the that's the whole reason I got into that in the first place. I'd been I'd been running FreeNAS at that point by like for like five or seven year I don't know, since I went from Windows home server to to Free Nas as I recall. So Which became true NAS to be clear. Yeah, the classic path. But anyway

Um, the thing that's happened though is we've done like our s our stuff has evolved. You know, I when when we started this, I was running true nas free nas on a Broadwell E Uh, like my old gaming machine, which was a Broadwell E eight core machine that was like using five hundred watts of power at idle with no hard drive spun up or something ridiculous.

And um our setups have evolved, our hardware's evolved, uh the the services we're running have evolved, and we haven't really talked about it as a whole in a long time. Uh so rather than make people listen to three hundred episodes and piece together the gradual evolution of what we're doing. But we thought it was useful to like run down the run down the stacks and kind of talk about it. Yes. Should we start with my server? Yeah, we can start with your server.

You wanna ping pong back and forth? I don't actually have a ton of other stuff, so I'm mostly just gonna focus on my server. I will just like briefly run down the other boxes doing stuff in my house at the end of this, but I've I've talked about those enough recently that I feel like just

Touching on that, uh, should be fine. But there's a lot of there's a lot of meat to administrating this big Linux box. Yeah, you you you but the ne the interesting thing is we we've both taken different approaches'cause

Like you've gone with one monolithic server. I have a bunch of stuff spread out over a bunch of low power devices. Right. And and you're a bit more appliance. Well, you're I think your your kind of VM host is just a a a bear operating system. We'll we'll get to that. Yeah. Yeah. Okay. All right. My current server.

Is a is an alder lake core I five, twelve six hundred K. I'll just briefly run down the specs here. I I got a good I've got a very good deal on that I five, like very good. Mm-hmm. Or otherwise I probably would have gone for like an I three, something lower power. Well but Go ahead. I was just saying having gone with the really low power solution, there's a

It's nice to have extra compute. Yeah, you know, I occasionally compile software on there and stuff like that and it's got a you know, quicksync and anyway anyway, okay, so it's an I five twel twelve six hundred K. It's there's sixty four gigabytes of ECC DDR five. in there. That's the ECC's error correcting RAM. I had to get an Asus workstation board with a W six eighty chipset.

But that was expensive. Uh it w actually it was it was about three thirty, which is very cheap in that world. Okay. I wouldn't have gone for the E C C until Asus rolled out that lower lower end workstation board. Uh is the reason I went for it. And the RAM was barely more expensive than regular RAM at the time. Also, this was three years ago when RAM was

a commodity like a dirt sheep. Yeah. Yeah. Those are good days. Yeah. Ton of hard drives in there. I've talked about it before. There's eight hard drives mirrored in the main pool. Uh so there's a so only four drives worth of space, which comes out to sixty five terabytes. Good god. Total. Well yes, yes, I have been amassing Black Friday hard drive deals for a couple of years, and like I said, it's too much now because I

Don't actually need that much, it turns out. But anyway. Um there's also a couple NVMe drives in there that I use as scratch space. What do you mean scratch space? Like um kind of anything anything and everything. Uh like I record

Like when I record videos in OBS for of streams or like when I record the Ramblecast, which I do a video of every week, like it records to that rather than the spinning drives because the because no access time and I don't want to worry about like Anything seizing up in the ch in the IO chain. And you're directly connected from your desktop PC to the NAS with like a high speed connection. I'm over this forty gigabit connection to the NAS, so it's pretty

pretty robust for sending large amounts of data. Although frankly, that's that even that amount of data could go over a one gigabit link pretty much. Well just fine. You say that, but like Um, I find that OBS is really sensitive to network latency. So that like if you if the machine for example, when I tried to run all of my my net my OBS setup stuff off of the the NAS.

Booting up OBS, sometimes it would just be weird because it would like lat the the drives would be spinning up or it would take a second to find a file or something, and it would completely It it didn't like that in ways that caused unpredictable bugs. So Yeah, I've I've never had an issue with it, but like I also use that

scratch space. So those those are two terabyte NVMe drives that are they currently are mirrored because I basically decided that everything in this machine should be mirrored since I use it for work and other

kind of critical stuff a lot and I just don't ever want a drive failure to take anything down. Yeah. Bec because there's not a lot on that mirrored volume, MVME volume, that is super duper important, all those re those recordings are pretty important. But I have considered striping that volume.

Which would make it four terabytes'cause striping is no redundancy versus anyway, two terabytes, still plenty of scratch space. What if you added a third NVMe and then did parody so you could have both do that? Yeah. I could do that. Think about it. Now that you mention it. But I I'll also use it for things like if I download like um

Uh like I downloaded a ton of PlayStation ISOs for Mr. Stream a couple years ago. Like a ton of them. And it's nice to have an extra high speed drive in there to like, for example Like if I I downloaded let's say like a couple of terabytes of of disk ISOs.

Uh reading those and writing them to decom because they're all compressed. They're all like seven zipped or whatever. Reading and writing those to decompress them on the same drives is pretty slow. Being able to de like have an intermediary drive in between. Yeah, it's way, way faster to have Have that extra drive space on there. So anyway. It's too slow going across the gigabit to the to the network.

Um lastly two SATA, two cheapo twenty dollar uh SATA SSDs in there that are mirrored as the operating system volume. Okay. So that's your route and then your hard drives. Are your hard drives the is the big stack of hard drives in a ZFS precision? Yes. Yeah. So it's all a big ZFS pool. Um. Okay. Okay. So why did I stop using TrueNAS? Yeah. Hey Brad, why did you stop using TrueNAS? Well, I'm glad you asked'cause this is this is kind of what James was getting at. Like

The the simplest way I can explain it is that I just kind of got tired of chafing up against the limitations that an appliance type operating system puts on you. And like those to be clear, those limitations are there for a reason. In fact, they are the whole reason that an appliance style thing works. It's the strength of the appliance style OS. Honestly. So when you say appliance you mean stuff like

um like uh Proxmox or Unraid or TrueNAS or it's like something that is like you install it and then you manage it probably through a web interface or an application rather than like a a Linux console. Right. And and crucially uh

It is a layer, those things are just layers on top of the same technologies we're about to talk about, the same software stack. It's just that like you said, they give you a nicer interface, they unify everything a little more, and crucially, like I said, they put up a bunch of guardrails that you're not supposed to go outside of. because they need they need they need a known state for the system.

in order for everything to continue working together. Right. Well and and so you can do things like update and stuff like that. That's exactly it. So like for example, like TrueNAS You're not supposed to go tinkering around with the underlying operating system because in even in the best case, like a a true NAS like system update will just wipe out all the changes you made. Yeah. In the worst case, you will break something irreparably and have to start over.

Yeah. I like this w this was I think the one of the things I actually really liked about TrueNAS. Yes. Yes. It is a hundred percent just a philosophical approach. Do you want to really do you want to be a mechanic And like be uh being on your back under your car tinkering with carburetor and changing the oil yourself, or do you want your car to just go?

is kinda I guess the best metaphor. So I d I dropped true Ness, um, this is while we're talking about it. I I dropped true mas true Ness for at the around the same time but for completely different reasons. Um my uh hardware star started getting janky and uh my Broadwell machine started getting wobbly. And when I was looking at building a new machine for that.

I was like, oh, let me look at the power requirements for a desktop PC and what I'm paying for electricity now. And I was like, oh, I don't want to do that anymore. What's what's the less power consuming option here? Um, and I went I went out and bought a Synology Ness, honestly,'cause I was like, oh, this has a this has a decent size decent uh like

four core Celeron or something in it. It has QuickSync. It has all the things that I thought should let it run most of the applications that I was running on that Broadwell machine.

Um, without a whole lot of uh may maybe maybe things like compiling updates and stuff would be a little bit slower, but uh the day-to-day operations would be about the same. Now, in practice, not true. The selleron the the NAS that I bought was a little too underpowered for that. Um but The the in terms of like power consumption for serving files. It's way, way, way, way, way less money every month.

Sure. Uh for the like to the tune of like twenty bucks. Yeah, like I I would guess the Synology is not using that much more power than it takes to spend the hard drives. Obviously there is overhead to run the electronics and the CPU and stuff in there, but like it c contrast with my twelve six hundred K that's probably using

It's probably a hundred watts at idle, that'd be my guess. Or I'm thinking even without the drives. Without the drives, yeah, it's it's it's not insubstantial. It's there's a reason I talk about wishing I had solar a lot. So the the Celeron uh it's a J forty one twenty five, which is a mobile Celeron in this one, uh Gemini Lake refresh, which I think that's after Alder, but I can't remember, is a 10 watt T DP. So it's like wow. It like yeah.

Yeah, versus versus mine. I don't know what it is at Idle, but I mean the I think that twelve six hundred K is like a hundred and fifty watt T D P. I mean it's not running at that all the time, of course, but Yeah. Um okay. Should we get into the software stack? This is the thing James James was actually asking about. Okay, so I basically just got tired of not being able to like do whatever I wanted on a har on a machine that I owned, you know.

And aga and again, if you're if you're if you just want a thing to work, the appliance stuff makes perfect sense. But I was constantly going like, Man, I wish I could install this and use it, but it's not part of what they have integrated into this thing, or like, Oh, I wish this scheduling thing worked better.

Et cetera et cetera. Or like there's a new there's a new piece of software that I want to try. It's just you couldn't do that on T. You know what it really was was their their um they're scheduling for like backups and scrubs which Scrubbing in ZFS terms is like checking the data integrity on the drives. But like their scheduling was not as robust as I wanted. Like

For for me the big challenge with the BSD'cause the thing about the at the time that we switched over, at the time I switched over at least I was running TrueNAS the BSD version. Yeah. And which which they are like killing off. This is we don't need to get into this tangent, but they are basically killing off the FreeBSD product at this point. I mean, it makes sense. Yeah. Because the problem with it was

Stuff would mostly work except for when it didn't. Right? Like you could you could probably cross compile a Linux a piece of Linux software for it. But it would be janky or weird or you'd have to like Jump through a bunch of hoops. Then that's extra overhead, yeah. Which which was not great either. Although to be clear, the scheduling thing I was talking about is is true NASA's like UI middleware and the built in one, yeah. I was just like

I I wanted to be at like, you know, I wanted to only scrub on the first Sunday of every second month or something like that. And they wouldn't let you do stuff like that. And then I started thinking, like, you know, that wouldn't be that hard to just script that yourself if I had the liberty to do that. I've just used the word liberty. This is getting

It's all about freedom, Brad. This is getting too ideological, I'm afraid. But anyway, okay. Okay. So I'd I I decided to go bare operating system. I I actually was going to move to FreeBSD first because that's what I was used to, but it but had basically zero support for the Alder Lake at the time, three years ago. Of course. And looking it up recently, they are only now starting to get like full heterogeneous core scheduling support.

integrated. Like FreeBSD is cool, but it just does not have the development support to to even come close to catching up with Linux in terms of hardware. Yeah. So that's what drove me to Linux. It was like, all right, it's time. This is what the world runs. This is what all the software is made for. It's time. I I mean look there's a real argument in this space for doing the thing that's

Like if you want to learn a bunch of stuff, then you should do the thing you want to learn, right? Yeah. But if you want to have something that's a usable functional computer, there's a real argument for doing the thing that everybody does. Yep. Yep. You're not wrong. You're not wrong. What are you doing for file system? Uh well so it's it's still ZFS. I'll so I'm I'm running I'm running uh Debian stable.

Debian Debian thirteen Trixie is the one that just came out last fall, so I'm I'm on that now. Uh I picked Debian because it's stable. It says it's stable right there in the name, but like it is it is it is known to be the most like boring, slow, doesn't update often and everything is supposed to work together properly and tested well distro out there pretty much.

I think in this case that's a strength, not a weakness. So I I I am very much in the camp of wanting a server to be boring and and stable and reliable. Uh that said, I've been through This is my third or fourth third d third Debian, I guess, since I'd made this move. And like every version of Debian, there's at least one major package that they don't have packaged or is way out of date.

That I really wish I had access to and don't. So there are trade offs with everything, of course. I could move to a more aggressive distro. I just have not. Um yeah, so so I am still using ZFS because that's what I was used to from true NAS. ZFS is the big enterprise. You know, checksumming file system that

prioritizes data integrity and stuff like that. We don't need to get too deep into that for the moment. Um but so you can install ZFS straight from Debian repos. That's not a problem. You do have to like agree to, hey, this license is not GPL. Are you okay with that? But that's fine. Why why did you You did you say that you installed ZFS on the root? So so yes. I somewhat controversially in the Linux channel on our Discord, I am even running ZFS as the file system for the root volume.

I don't know about that, Brad. That requires a lot of jumping through hoops. Probably the scope of this episode, frankly. Yeah, I I mean I get So the benefit of ZFS is it's similar, it's a journaling file system. You can roll stuff back. You can um Like if you if you Or or it's copy on right is actually the is is the the term for for what it does. So actually maybe to put it in terms that say like dual boot diaries listeners might understand.

ButterFS was kind of made in the uh spirits of ZFS. ZFS came out of Sun like twenty years ago before Sun got gobbled up by Oracle. Uh ButterFS is very much doing the same type of stuff that ZFS started. In terms of snapshots and and uh check something and volumes and stuff, sub volumes and stuff like that. Logical volume management, the whole thing. Bcash FS, which we've talked about, is also very much trying to implement a similar feature set to ZFS like

It kind of abstracts out a lot of file system stuff. Yeah. So that you can do things like rollback to previous versions of of files or rollback to a previous state of the entire file system. Stuff like that for people who don't know about this. Aaron Powell It's also and again this is again a little out of scope here. It is an entirely vertically integrated file system and volume management tool. I don't know what that means.

Traditional file systems were like you need you need a RAID management layer underneath the file system layer. Oh right. These these copy on write file systems do everything. They handle all the partitioning, all of the volume the physical volume management and the file system stuff on top of that. And the distribution of data across the physical disks. Right. Yes. So so you can do things like just have a bunch of different size disks.

disks jammed into your TFS pool and they work theoretically. Yeah, yeah, yes. There's there's management there. I mean, so this is one of the things James was asking about because this is one of the things that TrueNAS both abstracts for you and also makes relatively friendly

With a web UI? Uh for for in fairness, Unraid and Proxmox. I think doesn't. Unraid got ZFS fairly recently in the last couple of years, I think. Proxmox I believe does have ZFS support of some kind. I'm not an expert on that stuff, but but

Yes, to address the point, I have had to learn a lot about the ZFS command line tool chain. I mean a lot. Like perhaps more than I I shouldn't say more than I ever wanted to know because I love knowing this stuff. Like it's kind of the whole reason I'm here. Yeah. But I Yes, you will spend a lot of time understanding understanding how this stuff I mean, it's cool if you enjoy this stuff because I think that the I think that the the tooling around ZFS

That's provided at the command line is incredibly elegant and does some really cool stuff, but but it is a lot of learning. See, I looked at this, I looked at the work you were doing on this and I was like, I'm gonna go buy a thing that I can configure with a web interface. Totally valid. Yeah. Uh you I mean you you you have to be down for this as a hobby if you're gonna go this route. Um running running that root volume on ZFS required me to basically do a manual install of Debian.

Uh it's it's very similar to the Arch install if you've ever done that. It's all command line based. It's like the old school Debian install, right? Yeah, you basically have to build up the kind of ZFS volumes manually from a like a live C D and then manually install everything. Um Having having done a fair number of Linux installs in the last year, six months, let me go ahead and tell you.

I don't like the partition manager in any of the graphical Linux installs'cause they don't give you granular enough control over anything and you always have to resize shit so that like

Hibernate works or whatever. It's one of one of the reasons I kinda washed out of tinkering with desktop Linux. God, it's been almost a year now since I was fooling with that. But yes, like Like the Fedora installer, for example, made way too many opinionated choices about how to arrange the ButterFS volumes in a way that kind of like defeated what I was trying to do and I kinda got fed up.

It's like we need to make Brad O S. Like I guess how you do this. I guess so, or I really should just run through Linux from scratch, but I I'm pretty comfortable with all this kind of manual partitioning and stuff at this point. Um okay, so I had to do the the manual Debian install, which meant that I had to like choose how to configure my network and choose, you know, make the UEFI boot entries myself.

Like you're kind of picking everything. There multi as I've said many times, there's like six ways, for example, to configure your network interface in Linux. So you kinda Just need to pick one and go with it. Yeah. Um that's a like no wrong choices situation. There's like Like you can't

You Yeah. Well you know, this is a case where I would e I would say, for example, the Debian handbook makes some recommendations about like, How are you using this? If you're on desktop you should probably use network manager. If you're doing a server, system D network D might be better. That's what I'm using or or that the In that case, I think they describe it as the modern headless network configuration.

Um people really don't like system D stuff, Brad. I don't know if you know that, but that's a different topic for a different day, I think. It is. Okay. Um and then like I said, I I'd have to maintain the like the boot entries myself. I we've talked about it before. I use a ZFS specific bootloader that maintains compatibility to make sure that

There's a wait. There's a whole thing Are you doing secure boot there too, or are you not? Yes, I am doing secure boot there, which means that I have to so I generated my own secure boot keys that I enrolled into the UEFI. And I have to so I have to sign it's called ZFS boot menu is the bootloader I'm using. And yes, I have to

I have to sign every new version of that to make sure that Do you do that manually or do you have hooks that do that when a new kernel gets pulled down? CFS boot menu, I do that manually. Oh god. But but I set up um I set up DKMS, which is the Here we go again. Did you know there are multiple subsystems in Linux for building kernel modules? Of course there are. There's DKMS, which uh like Debian uses, I think Arch uses, but then there's AK mods, which is used by the Red Hat style distros.

Anyway, those things can be DKMS in particular can definitely be configured automatically with your keys. Yeah. So anytime it builds a new ZFS module, for example, I have it set up to just sign it with my key automatically and that's fine. I don't have to touch any of that stuff. But the thing I was gonna say real quick

Uh and I'm I'm making it more complicated on myself by running ZFS root, to be clear. But yeah,'cause like like theoretically if you had installed on like a Butterf S or or EX3 or something, you just Do all the you'd set up the machine, do all this stuff, and then you'd flip on the ZFS side. You wouldn't have to do any of this business. I could totally be doing like an MD raid and X4 setup for my

Boot volume, which is still mirrored and redundant, but using like stuff that's in the kernel and not have to worry about it. If you were doing this again, would you do that or would you do the ZFS? I thought about I thought about moving to it. But the reason the reason I clung to V ZFS is because I'm already using it to manage all the other volumes in this system and I kinda I kinda just wanted to keep it all to one. style of

Drive management for everything. But I have thought about doing that before and and that's why I was so bummed that B cache FS got pulled out of the Linux kernel recently. Because I was thinking maybe one day I'll move my boot volume to that because that is in the kernel, but now it's not. Scene drama.

Yes, kinda yes, that's definitely some scene drama. Um to to give a very quick illustration of how this can get complicated. Uh-huh. Uh because ZFS is outside of the kernel, they have to track kernel versions, meaning like every version of ZFS they put out is compatible with only a range of kernels.

Typically up to the newest kernel because they are pretty good about staying up to date. But occasionally the point is occasionally a Linux kernel will ship before ZFS is ready to be compatible with it. So it is possible, and this has happened to me before, it is possible to update your kernel beyond the point where the kernel can work with the file system you've got your operating system on.

So then how do you fix that? You C H you boot off a thumb drive and C H root into your system and journal? Okay. Is typically how I've done it. Now do you do you have snapshots set up for your ZFS snapshots set up that are integrated with your bootloaders you can just change back to the previous one? So I I think maybe that's a good way to focus this down so I don't ramble forever here is to kind of go through the things TrueNAS does for you and sort of talk about my equivalent.

Yeah,'cause the nice thing about TrueNAS is if you had a bad update, you literally on the boot menu could just pull down the last like five states of the machine, however many snapshots you had. That's and And roll back to how it was yesterday. Yes, that's that's one of the things TrueNAS manages for you very well is is doing auto snapshots. You do have to tell it to do that for some drives, but like the system drive in TrueNAS, it just does snapshots every time you update. Mm-hmm. Um

But but yes, that's one of the things I had set up on TrueNAS was just having its snapshot my every every pool in the system gets a snapshot every hour, every day, every week, et cetera. And there's policy for how often to call those. Um those don't use any extra space to be clear. That's kind of how copy on write file systems work. Anyway, I'm using a tool called Sanoid, S-A-N-O-I-D. Which has been around for quite a while. It was written by Jim Salter, who is a a sometime contributor to art.

contributor to Ars Technica. I don't know if he still writes for them or not. I don't know. I I've always tried to avoid the Noid. That's my understanding. Yeah. Uh but it seems like you've embraced the Noid. Any anyway, their Sanoid is the most robust tool I've seen out there for ZFS auto snapshot. snapshots. Uh it's it's command line only. It's uh

It is packaged for some distros, but I just went and installed the dev. In fact, I think you build your own dev from the GitHub repo and then install that.

Do you does it and it it triggers a snapshot anytime like uh you pull a package down from from the Debian repository or something? Uh y yeah. Yeah. Uh and it it's all it's all policy driven. You're you're I I sh it's probably worth noting here. You're in in my setup or a setup like this, unless you set up some kind of web UI, you were just configuring everything. From a terminal in text files and config files. But yes, it's it's policy driven where you can tell it like

Snapshot this pool every hour, keep forty-eight hourly snapshots, et cetera, et cetera, you know, by pool, you can configure that stuff. Um Stanoid has an optional subcomponent called sync oid. Oh. S Y N C Sync Oid. Okay. For replicating the snapshots that it's auto snapshotting to other volumes. Oh across a across the network or whatever. Or locally.

I swear to God, I'm gonna finish setting this up one day. It it's the Raspberry Pi down the hall with the the four drive enclosure bolted to it. Uh I'm I'm gonna say, like, that's one of the things that was one of the appealing features for me on ZFS when I first started down this path is that I could like set up another machine that's plugged into at the time the w the tested office, right? And

And I I just have an off site backup of my stuff that happens automatically. Yes. No, I don't I don't bother with that anymore. I mean it it like all of this, it is a lot of work. But the workflow th typically the workflow that I'm trying to to achieve here is

I never think about any of this stuff once it's set up. I just use the server as is. You know, I put things on there, like podcast recordings go to it, whatever. At night, I've got it or theoretically I will have it set up so that every n every night it just syncs whatever that day's. additions or changes are to the volume down the hall. Uh and I I basically never touch any of it unless I need a backup, right? That's the goal anyway. Um I've also got the snapshots uh

on the NAS setup as w what used to be called shadow copies in Windows. I think they've changed the name. Shadow volume copies is what we used to call them. Maybe that's it. Effectively Effectively I can get properties on any network mounted folder and it'll show me a previous versions list and I can just roll back to that actually saved my ass a couple weeks ago when I deleted some old Ramblecast recordings.

uh to to clear space and then like and that that one is on a six hour call uh schedule. So like two hours after I deleted them, thankfully, I realized I needed something from one of those old recordings and I was able to go into those into the previous version through the Windows File Explorer and pull that file out before it truly got eradicated. That's good. And now

How am I sharing these things to Windows? Yeah, I was gonna I was gonna say like what services are you running on this? That's that's the big question. So I'll kinda keep going down the line here. Samba is SMB is what I'm using to share. Okay. The the volumes on this thing over the network. Uh there are other options. There's like NFS, which is a more

traditionally Unixy way to share so yeah I use NFS for like Linux to Linux talking, but not so much for Windows to Linux talking. That's that's exactly why is the Windows support for NFS is very Rubinary. Bad is it sure. That's a gr that's a fine way to describe it.

So so the nice thing about NFS is it preserves permissions and lets you map like a user on one machine to a user on another machine or use Active Directory or something like that to to talk across both of them. Not Active Directory, but like a a a shared a shared

source of truth for which user is which user. Yeah, in in a more like n natively Unixy way, right? Like you you can do that stuff with Samba as well, but it's much more fiddly and and arcane, I find With Samba you usually crush down the permissions to be like the least common denominator that everybody needs to be able to do the thing you want to do.

Yeah, it's it's kind of a mess. Anyway, not a lot to say about Samba. Like you I I was using Samba or SMB in TrueNAS as well, and I and again I do it because Windows is good with it. Mac OS is also good with it, so I can mount those It's volumes for my MacBook as well. It's funny. I often use AFP on my Linux machines to talk to the NAS when I'm on desktop because for whatever reason, like Nautilus and all the file managers that I've used in the desktop Linux.

almost always see the AFP shares on my Synology NAS before they see the the win the S and B shares. Interesting. Um So yeah, I I d I don't know why that is. It's just a weird like side effect of opening up the network browser and being like, oh, okay, so there's two things here. It's the same machine both, but this one always pops up faster, so I use that. Uh what was I gonna say about oh I I've got all the advertisement turned off for that stuff, so I always just mount stuff by IP.

I have other people using my computer. Yeah, that yeah. I don't I don't have that concern. So Samba, not a lot to say there. You you configure it by managing SMB.conf, like it's relatively straightforward, although there's like a

ton of complexity to advanced Samba configurations we don't need to get into. Yep. Uh okay. Services and containerization. Yeah. Is like probably the thing that I started to say probably the thing that was the most problematic on TrueNAS, I don't think that's quite true, but it was the thing that on BSD you only had uh jails.

For well and then they also had plugins, but plugins were a little fraught. Yeah, I never even touched the plugin stuff'cause I people were like, Oh, their Plex plugin is super outdated or doesn't work. Right or blah blah blah. Yeah, the the bigger problem with the plugins was that they would always have some sort of weird limitation. So if you were doing something that was like Um

that that had relatively low overhead and relatively low interconnection with other services, it was usually fine. Um, but anytime you wanted to do anything that went beyond like the default, then it was a problem. Like, for example, the Plex plugin was great. Figuring out how to get your Plex database out of that Plex plugin. Uh, the Plex database contains like the list of movies, the number of times they've been watched.

um the users that are allowed to attach to your server, all that kind of stuff. Uh a at a giant cache of all the like the art and the the art that gets displayed in blacks around each movie or TV show.

Figure out how to get that data out of the Plex plugin on FreeDAS was a nightmare. I bet. Man. I mean it's it's not easy to get it out of a manual install of Plex. No, it sucks. Like if you've ever migrated Plex from one operating system to another, like their support page for doing so is like twenty steps long. Yeah. And you have to be kind of careful about it. Have you done it? Yeah, I've done it three times. It sucks every time. They are very specific about like

Okay. Like before you log into the Plex on the new machine, you have to log out of the one on the old machine. Make sure you don't clear the trash on that one though. We copy these files over, like do this, do this. Like it's it's weirdly sort of um fragile, it feels like. So I have that set up now so that my Plex database is just as a folder, uh a a network share on the NAS. And um

When I connect to it I have to reverse engineer how I did that every single time'cause it's a huge pain in the butt. It sure is. Uh okay. So um my understanding is since TrueNAS has moved to Linux, I mean obviously kinda like me moving to Bayer Linux, that has given Trunas users like way more options for the types of sort of containers and plugins they can use. Are they still using jails?

Uh n well no, not on L Linux. That's a boy, that's a big topic. Uh so ja well ja okay, so for folks who don't know, jail's basically Um, the the short, short version of what a j a BSD jail is, we've talked about this before. It's really cool. It's my favorite thing about BSD. It's the reason I use TrueNAS for such a long time.

Uh it basically lets you spin up a virtual what looks like a virtual machine on your network that's essentially it has its own IP address, so you can give it its own like uh DNS entries, stuff like that. And it takes The system files from your main OS

And lets you use them as if they were in a virtual machine without the overhead of a virtual machine. Yep. Is kind of the the the short short version. So the re the reason for that is'cause they're running on the same kernel as the rest of the machine is, so they don't use extra memory or there's there's

You're not you're not siloing resources off to them that only they can access. They're kind of there in the that's that's the big one, is that you're not it's not like running a virtual machine where you say, Oh, okay. I have four cores to eight cores total on this machine. I'm gonna devote two cores entirely to this VM

And then the main OS won't be able to use them anymore. It it it let you resource share much more effectively. It's it's a trade off of lower resource overhead for less security because a VM is much more sandboxed and less and is is is is much more impenetrable than

It also means though you can access the file system a little bit easier. So you can you can you basically you'd say, Okay, I want these subvolumes of your ZFS array to be accessed by the jails and um For like a corporate user who's really worried about security, maybe not the best solution for a home person who wants to make it easy for their Plex server to see their movies.

uh piece of cake. Yeah. So maybe I'll I'll I'll I'll go in order of how I came to these things chronologically,'cause that might make it a little easier to explain and and build on what we just talked about. Um

So in so in FreeBSD, I would say like jails are extremely secure because they are built in at the kernel level. Mm-hmm. Like they are they are what people would refer to as a first class feature of the operating system, meaning they've, you know fully integrated into the kernel, like fully security audited.

Like I don't think a jail breach has ever been demonstrated before. They've been around for a really long time and they're they're battle tested. Twenty something years or whatever. Linux does not have an equivalent at that level. Yes. Like l like the Linux kernel itself does not expose a comprehensive containerization.

pattern or whatever you wanna call it in the way that FreeBSD does. Now there are there have been Linux containers for like fifteen years that are built out of first class Linux kernel features, like And I'm getting into stuff I can kind of only somewhat understand like namespacing and and control groups, which are Well, explicitly. From from a practical level, the solution that solves the same problem in a different way on Linux is Docker.

Right. Well, yeah, I mean that's what I'm getting to. And that that is largely what I have moved to, but I I'm like I said, I'm going in order from I kind of Oh when I when I got to the Linux NAS, I basically was trying to sort of replicate everything that TrueNAS did. Yeah, that's a bad idea. One to one. Well it was fine. It worked fine for a while.

For a while. Oh, it dude, it it always would have worked fine. It was the management it was not it was not like it was broken, it was just that the management was ultimately Much like on BSD, effectively you're managing a whole little OS for every sys service you're running. I I would argue that onerous management is not working fine. Yes. I mean that's kind of where I got to eventually for sure. But so so um the the classic Linux container technology is LXC.

Or or I guess Lexi maybe is how you're supposed to pronounce that, but it but it is three letters L X C. Like that that is the old school Linux container technology that was kind of derived as a BSD jail equivalent out of kernel features in Linux. I used that for a long time, like two years maybe after I moved to Linux. Like the tooling around that stuff is very bare bones. It was a huge pain in the ass to administrate.

running running those containers in a in a secure way, like a rootless way, because I wanted to not run those containers as root. Yeah.'Cause if you're gonna be running services that are exposed to the outside world, that's a vector for

That then you're giving those services root access to your machine, which was bad. Like you don't anything you want anything that's running pointing to the outside internet in my book, you don't want running as any kind of privileged user because if those things are compromised, they get a lot of access to your machine all of a sudden. Um

Without getting into too much detail, the LXC management was extremely fiddly. The the command line tools were super bare bones. You kinda had to figure everything out yourself. Running those containers rootless was a lot, although it did help me quite a bit for where I ended up.

Which is after a couple of years of that I got tired of basically running like like I was literally having to like update a whole different Ubuntu um Yeah, that sucks. Install for every service I was running. Like I I I was gonna say the like the Thing I like about this is that using the LTS versions of U I'm using a L L T S version of Ubuntu, not Debian, but like

It means I don't have to update it very often, right? The updates are relatively infrequent. They're generally less scary um than like a rolling distro like I'm running on my desktop. And for things that I use all the time and are important that they not break, it's it's really nice to like log into that machine and be like, Oh yeah, you don't have any packages to update. Yeah. If I had to do that for five Linux machines, I would kill myself.

I think at some point I wrote a l a tiny little wrapper script. I mean we're talking like three lines or something to just update all of them at once. Which ballsy. Not you know, not difficult. I I should point out like one cool thing though about that setup. is on FreeBSD the jails are just another jail of the same FreeBSD host that you're running on. Yeah. With the Linux stuff, because there's so many different distros, I could be running Debian stable as the host.

But you can kind of pick any distro that they and any version of a distro that they offer an image for as your containers. So I was running like much newer Ubuntu server images as my container. Uh distros on top of and you can totally do that. It's fine. And you were able to p do things like pass through access to quicksync and stuff like that? Totally. I mean that that took work for sure. That yeah, like I had to learn more about Linux Terra.

Yes, like yes, exactly. Render render render D one twenty eight was getting passed in. I so dude, we could s we could this could be a two hour. Render D's what? We we could we could sit here for two hours talking about this stuff. I I had to get pretty well acquainted with the way w um Linux handles hardware devices as files to do that, because again, LXC is like it's well supported, but it's not well

Tooled, I guess is how I would put it. Like the like the tools again are not friendly at all. You kind of have to figure everything out yourself. So like document documentation good? It's it's it's not nothing, but it's not enough. Okay. F okay, as a specific example, the the technique I ended up using for passing the quicksync device into my Plex container. Which I did all the work for before I found out that Plex requires a Plex pass.

To use hardware transcoding, although I was also running Jellyfin, so it was at least useful for that. Yeah. Uh the solution that I ultimately used, which I was quite happy with for passing that Quaksync device into the container. Is something I just dredged up from a random LXC forums post from three years before. To to be clear. When you ask, is the documentation good? Like it's the bare bones stuff is good, but beyond that, if you want to do anything advanced,

You're digging through forms. Oh great. Okay. I don't like that for what it's worth. Yeah. Yeah. It's a lot. So where'd you end up? Uh where am I now? I have I've mostly moved off of LXC. I'm not running the LXC tools as such anymore. Uh at first I can't remember what order I went in here, but I'll just go.

Let's just talk about Docker. That's the thing everybody cares about here. I mean, that's that's the right answer for this problem, it seems like, most of the time. Except I'm not running Docker. What? I'm using Podman to run Docker images. Well that's that that's still running Docker Images. Yes, it's still running Docker images. The the images are compatible. Yeah. Um is Podman a portainer? It's that's like the that's the

Is that the the the the lower overhead version of Portainer or is that a different thing? Not at all. It's it's totally separate. Podman is a is you could look at it as a different implementation of what Docker does. It's a different container backend. But you use like the same kind of Docker compose files or whatever? No. Oh. You could. There is there is a Podman Compose tool that you can use effectively the same YAML files if you want it.

But I think I want to say even the developers of Podman Compose basically say like, hey, this is here if you're coming from Docker and want to uh use the same style of YAML configuration you're used to. But There are better ways to do that now. Podman came out of the Red Hat slash system D slash like that whole constellation of of products and services. Okay. Let's say like it's the standard Linux thing of

Somebody comes up with a way of doing something and then somebody else says, You know, I see X, Y, and Z problem with that. I think I can do it better.

Uh although again, it c coming out of like the Red Hat world, it is very well robustly supported and developed. Like it's not this is not Podman is not a fly by night thing at all. Um Podman and Docker are two different ways of doing the same thing, which is downloading images, quote unquote, of applications you want to run that are fully self-contained, sandboxed bundles of like the binary, the core binary of the service, all the libraries that are required to run it. You know, kind of a

Well th yeah, the idea is that it gives you the the machine state that you need to run the service that you want. Now, ironically, it's doing effectively the same thing and in fact using the same kernel technologies by and large as the LXC containers I was doing.

It's just a friendlier interface on top of them? Well no, what it is, it's just bundling up only exactly what that application needs to run, unless it's a poorly made image and somebody puts way too much stuff in there. But by and large, like well well authored images and and

people on the Linux channel on our server definitely going to scream at me at some point in this conversation. But that's fine. My understanding of it in the little bit I've looked at how you build up Docker images is like you really only want to put exactly as much as you need for the image to work and nothing more in there. Yeah. That's right. So like

Docker images are also based on specific Linux distros typically, but they're so stripped down that they're not full Linux distros. My understanding is you could run a full Linux distro inside a Docker container if you wanted. Yeah, y people run like um you can run like a a desktop you can run a full desktop inside a Docker container if you want.

But it's a bad like I wouldn't do that. What is um there is a is it distro box, I think is the tool that I I see recommended. There is there is a tool that can use either Docker or Podman as its back end that will let you run. Uh yes, it is distro box. Like you could you could try out other distros in a Docker slash podman container type format.

Is so that's yet another way to to do the thing I was doing before. So what are you running in all these podbands? Uh it's it's the it's typical media serv I d actually don't run that many. Uh it's typical like media service stuff like Plex, Jellyfin, Own Tone. Phone Tone is your MP three server? Well it's it's the airplay server. It's the thing that um it's the thing that exposes through the network as an airplay like uh

Trying to like what are the products they make that can do this? Like in the back in the day it would be like the your iTunes running on your Mac could would show up as a library to play from, but these days it's more like what? Uh like music does that. Or or I mean from Apple though, like is does the home pod do that? Is that what the home pod is for? You just ask it to play something and it plays it, you don't have to think about it.

At any rate, Onetone is a way to look like you have an Apple, like an iTunes library on your because I have a lot of AirPlay speakers in the house. So it's it's funny, like the airplay speakers that you're talking about are not current air like not the airplay speakers from

The old times. The thing you're talking about is that iTunes library sharing thing from like two thousand eight where as one day you turned on iTunes and all of a sudden you saw everybody else's iTunes in the office and then you could steal MP3s from them. Well, could you actually really get the files? Or just stream it? Of course you could. Of course you could. Yeah. But um Okay, so but real quick, I mean, do we want to get into why I'm using Podman so Docker?

No, it's fine. I mean, that's a big thorny question. Just say one way or the other. It's a yes or no, Brad. I like it better. But I I've I've barely touched Docker. I have like literally used Docker for about thirty minutes ever. Like I just don't have a good frame of reference that

The reason my understanding is the reason Podman came to be is because the people who made Podman again, people in that Reddit or not Reddit, Jesus. The R Red Hat. My my brain is so poisoned. The Red Hat, like system D kind of world. looked at Docker and felt they were not Docker was not taking it seriously enough to let you run containers not as root,'cause again, like we said, running services as root is

Is is dodgy from a security perspective. Well, Docker has since my understanding finally belatedly got around to more or less fixing that problem, but I think I'm really speaking out of turn here. I think out of the box it's still probably running everything as root unless you go out of your way to change that, but I don't quote me on that. I don't I don't know about that. I don't actually pay attention to that usually.

Anyway, very briefly, Podman is basically built from the ground up to let you run containers as unprivileged users, where Docker kind of I think that had to be retrofitted on. Okay. That makes sense. Is basically the selling point. Podman also doesn't run a demon, meaning like a like a a monitoring process, supervising process in the background. Did you say demon? Yes, I did. Don't you mean daemon? No, I don't.

It says D A E M O N right here. That's right. Look it up in the dictionary, man. Not to appeal to authority here. Opposite of the nightman. Not to not to not to just appeal straight to the dictionary authority, but Um, if you look it up it says archaic spelling of demon. I don't know how I feel about this, but I'm gonna let I'm gonna let it go. I'm gonna let I'm gonna let the chat handle this. Dude dude, the free PSD mascot is a devil. Yeah. There's a reason for that.

I'm pretty sure I'm pretty sure. Damn. Anyway, whatever. Whatever. Podman doesn't run a uh like a supervisory process that oversees everything else. Yeah. Uh like like Docker security stuff. Um So I I I run all of my containers, all my podcast podman containers under a kind of a nobody unprivileged user. Okay. And then we don't really need to get down this road too much, but I'm actually

Using a s a setting where the actual container processes inside the containers don't even run as that user. They run as a subordinate six digit user ID. Like ephemeral, ephemeral user ID. LXC does the same thing. This is actually like pretty standard Linux stuff, but So all of the like like my Plex process and my own tone process are actually running as user one zero zero nine nine nine.

How how do they have access to the do you have to give that user access to your files in those in the file system? Okay. AC ACLs. Do you remember when we uh interviewed um uh Jeremy Allison, I believe, was the Samba maintainer, co-founder? Remember, remember that? Do you remember him talking about Ackles? Mm-hmm. So I'd I had always called some Ackles. He called them Ackles and that's explains a lot. Uh you know, he's been around a while, he's got more authority in this domain than I do.

Yeah. I I always said them ACLs, but ACLs are and this this might be something that comes up on the dual boot diaries at some point, access control lists. Uh-huh. They're ways to give files other subordinate p um permissions to the main ownership permissions. Oh yeah, it's like the it's like the advanced permission stuff in Windows and T Find yeah, like Windows and N T F S has a similar in fact N N T F S has ACLs, like it's totally a cross it's

Anyway, so I use ACLs to basically just give that give that six digit UID uh access to the directories it needs and nothing else. Oh see I just crunch everything down to user one thousand and chair across the'cause I'm sharing across the network. So I'm connecting across the network so I haven't It gets really weird and complicated if you don't do that. Yeah, I d to very briefly defend my kind of security maximalist position

A this machine's got like my whole kind of digital life on it from college on. B Theoretically I am once again going to point it at the internet at some point. Like I you know, I ran a Minecraft server and a little web server on it for a while. Those things are not currently running, but I want to start using

it for game servers and like a blog and stuff again at some points. But and I'm and I l like to be real, like I'm I'm just enough of a minor public figure and have had just enough contact with anonymous trolls in the past that Oh, I know I'm pretty uh a little bit of paranoia, I feel like, goes a long way here. That's why you have to VPN into my house to use my servers now. Like so so like the the point the point I'm making is the average person maybe does not need to be quite as stringent about

containerization and and siloing things off as I as I am. I mean I think my solution for that problem is to just put everything on a different machine. Yeah. Right. Sure. Um That is that's not quite air gapping, but that's certainly a more physical separation than having all the storage in the same machine that all the services are running on.

Yeah,'cause w with my serv with with the out outward facing services running on so I have the two machines. I have the the NAS and then I have the little B Link uh with the Alder Lake Celeron in it. And um b basically I do least valuable permissions for each of the services. So

Anything that lives outside my network doesn't get right permission to the NAS and I just enforce that via the the S and B shares or NFS shares, whatever I use. So even if one of your the services gets compromised, it still only has access to whatever network shares are mounted from there and not the entire system, yes. Exactly. Yes, indeed. Um okay. So Podman instead of Docker for for application service containers. I still do use LXC containers off and on.

They not not as much. They should have called those podman containers podcons, just for the record. Yeah. I think. That's my or podcast, maybe. I don't know. Yeah. Podman. Yeah, it sounds like a the kind of a crappy B tier Marvel superhero. Kind of. Kind of. Yeah. Um so so I uh like I said, I still use the occasional LXE like full distro OS container.

What I'm now using for that is a tool called Incus, I-N-C-U-S. Is that based on the demon incubus? No, it's a type of cloud. I believe the cumulinimbus incus. Oh okay. It's the anvil cloud actually. Just looked this up the other day. Do you know have you seen those clouds that form and look like an anvil? No. You should you should Google Yeah, cumulonimbus incus is the type of cloud. It's like it's wild looking. I've never certainly never seen one of these in person.

But I've never seen one of these in person. I think I have read about them. It's like a type of cloud that like levels off on top and just looks like a literal anvil. It's kind of crazy. Phil Phil Plate occasionally posts pictures of clouds because he lives in Colorado where they have good cloud. Yes. And uh yeah. So What Incas is is actually a fork of LXD. Have you ever heard of that?

Yeah, I think we talked about it a little while ago and I said L X D's what? I like made a face at me. We sure did. Uh LXE came out of Canonical, the Ubuntu people. Yeah. Uh it's basically a more advanced way of managing both LXE containers and VMs, virtual machines. Like actual virtual machines. Uh it's just using QEMU to run Virtual machines under the hood, I'm sure you're familiar with that.

Oh yeah. I know you guys have talked about it. I'm gonna ask a question though, just so I know so the audience knows who doesn't know. Okay. Hey Brad, what's Q E M U? It's it's lower level virtual machine management technology at Linux that hooks into KVM, which is the kernel. The kernel virtual machine hypervisor. Okay. Yes. That's cool. I haven't actually used that. I use bottles and stuff. I don't I don't use virtual machines. Uh I've I swear I've heard Adam talk about

Adam talked about different ways to manage virtual machines a couple of times. But yeah. And and some of those front ends, I don't know if bottles is like this, but some of those front ends that you're talking about on Dolboot Diaries will also be something that sits on top of something like QEMU. In fact, that's what Incus is. Like it's all it's doing Incus is just a f slightly friendlier front end to both LXC and Q EMU for VMs.

Bottles is just an easier way to package up uh wine MVP or w for w running wine apps. That's that's a totally separate thing. Anyway, anyway, uh Canonical still makes LXD, but there was other scene drama and the original LXD developers quit the company and forked it.

And it became Incas. Um Okay. So really the only point of that is it's it's kind of nice to manage your system containers and VMs from the same common command line interface. It makes sense. Yeah. Syntax is all the same. It's quite easy to manage. That's not even remotely the only way to do that exact thing though. Stop me if you've heard this one before.

I mean look there's an infinite number of ways to do pretty much everything in Linux man. Yeah, there's stuff like libvirt that I think is probably actually more popular than Incas, uh that does the same thing and a lot of other stuff at the same time. Anyway. Now you're reaching the part of the list where it's like there's just a series of letters that don't that like

Hey, here's the first four letters on the A row of the keyboard and uh and then uh four letters that look like a word but aren't. Do you know how hard it was to get Google Docs to let me just type ASDF into this bullet point and not have it auto correct to something else? Look. I just the new Google Docs autocorrect blows. They added AI to it and it's the worst thing ever. Notepad was autocorrecting my typing this morning.

I had to dig into the settings for that and turn that off. What are they doing? I'm I'm increasingly on the maybe I should just go back to one of the simple note clones or like an obsidian branch or one of those. We should do I I I installed the Obsidian app. I have not like spent a lot of time on that, but I I want to start

I need to get off of Simple Note and want to start syncing text files to something I control and so I Yeah. I need to put some time into like self hosting or or putting it in my VPS or something. The the problem is I've started I've been using Notion now for like three years and the the wiki structure of that is it turns out really good for like complex branching, many faceted notes like I like to make.

Yeah. And Obsidian just can't cool one layer. It sucks. That's like a proprietary notion feature, right? Pretty much? Or you can't you can't just port that to some other application, right? I'm sure like you could, but I'm sure somebody's done it, but it's not in like mainline obsidian for sure. Sure. Okay. So th very briefly here. I I use a tool called ASDF. Which is a is a multi runtime manager.

This is something that if if you're if you're running a server, if you're running like a big NAS like I'm doing and it's just Linux, like this might be something that would come in handy for you. Um What it is is a way of installing runtimes for kind of every programming language you can think of, Python, Rust, Ruby, Go.

A bajillion others. Um, there's another tool out there now called Why not just do that in the system packages? So so that's something you will you will discover if you start tinkering with that stuff quickly in your Linux distro, is that you really do not want to mess around with

the system Python, the system X, Y, or Z, the system Rust, the system it it might vary by language, but Python in particular is one where you will constantly in fact, I think Pip, when you try to install Python packages with pip, But with the system installed version of Python, it'll straight up warn you like, hey, do you don't you should not be installing

Pip pac or like s random packages into your system Python, you should use some kind of runtime manager. I'm I'm not well versed enough to know exactly what all the risks are there. But any anyway the the I use ASDF. There's another tool that I think is getting more popular now called Mies. M I S E Okay. As in Mison PLOS. Oh no. Yes. Look, man, I I get it. Look, naming stuff's hard. I'm not gonna judge. Naming an open source project is probably pretty difficult.

But I would probably be tempted to just name it like thing D at this point. I I like what whatever it is D. Look, I have a lot of the per respect for the person who's like, fuck it. I'm just gonna call it ASDF'cause that's the first four letters my fingers hit on the keyboard. I th I think it is kind of amazing that nobody ever called a piece of software ASDF until now. Yeah, it's uh it's it's like

Yeah, man, that's pretty good. Okay. It's pretty good. Any anyway, I that's something that might be something to look at. Um if you if you're going to run a bunch of little random services like I do in different languages, like it's it's nice to be able to just be like, hey, I just want to install a Ruby runtime and then use the the gem package manager that comes with Ruby to install this little utility.

Typically when I want to do something like that, I typically just grab a Docker container and do it in that that handles the the business for me. Like the benefit of using Docker, I think over I don't I don't know what how what the what the kind of um Like how how Many people are building packages for Podman, but like the you go to Doc No, so actually Podman images are the same as Docker images. There's no distinction there in terms of Oh

creating you're you don't make a Docker image and a and a podman image. I probably should have specified that. Podman is actually command line com compatible where they especially when it was new, they were just straight up like you can just alias Podman to Docker if you want and keep running the same Docker commands you've been running. It uses the same images, runs the same commands. It's it's just a different back end, but in terms of the

User experience pretty much the same thing. So if you go to like one of the Docker one of the sites that has like big lists of Docker images that people have built, you just grab the wow, that's amazing. Half half of my half of the Podman or half of the images I'm running with Podman are straight off of like Docker.io.

Oh, that's right. They're they're all the exact same images. Actually, sorry, there's one other detail about Podman that you will appreciate that I really do need to mention, which I think is like the reason to use it now, besides the rootless stuff, which is that the more modern versions of Podman integrate very directly with system D. Oh. So I don't actually touch the Podman command line interface at all anymore.

So you just start it from a system D call you make a system D file that logges if you're if you're familiar with what system D unit files look like, like a dot service file, dot timer, dot you know, dot mount, there's you know, there's a million types of s systemd units, but you write a.container file with the same syntax that you would write like a service file with. And then Oh, that's wild. And then it auto generates a service off of that dot container file.

And you never runs the Docker image. So you never even touch the Podman thing at all, like the the command line stuff tools at all. You just start and stop those services like you would every other system D service. Wow, so that's a that's like that's At some point in the in this year I hope the Wildcat Lake uh CPUs will launch and computers will still be inexpensive enough that I can afford to buy one of them.

And I'm gonna move my little B Link server over to one of those and rebuild everything from scratch. And and that I my initial thought was to just do everything on Nix, but maybe it's easier to do it on like Debian LTS. And uh and then do everything with Podman and System D. That's or Docker's still totally viable, like to be clear. Like don't need if if you'd rather use Docker, it's it's certainly I I just I happen to like

What I really like is is using one interface for everything. In fact, kind of a theme here, you know, like I use like using ZFS for every volume in the system. It's nice to use Incas for both LXC and VMs. And by the same token, I'm I've I'm running other custom system D services that are not Pod Podman slash Docker images. Some of those are just like

Like Blocky is just a Go binary that I download from the repo and just run directly via a systemd service, right? So it's it's nice to just like manage all of your system services with the same kind of general paradigm is is the other reason. How do you back this up, Brad? Snapshots? How do you back it up, though? Snapshots? Well, it's, like I said, I died.

Still have not gotten that that uh backup solution. Okay. Everything that matters, like like all the Podman config lives on a specific ZFS data set, for example, and that gets that gets auto snapshotted. But yes, I do need to be Replication at least save that to Dropbox or something. Yeah. It would be a bummer if that drive died. That's why it's mirrored though. Yeah. You know, a meter would have to fall on that computer. Yeah, or it's probably wiped out the m the CFS doing too.

It's on the list, I promise. Cosmic rays. I promise I promise the the the nightly down the hall replication thing is on the list, but anyway. Okay. That's kind of the software stack. That's in the this is all living in the server, just to be clear. Yeah, that's all on this big machine. That's pretty much most all of the stuff that TrueNAS would be doing for me. There there's one last thing I should mention which

I think this probably went without saying, but just to reiterate, like there's no web UI here. This is all SSH, this is all command line. Like you it's kind of on you to to interface with it and and kind of get your head around it. Um

Yeah, I I mean, I'm gonna go and tell you, I've I have been on both sides of this over the years and spending a few months learning how to use desktop Linux has made me much less Like like the the design of the command line interface, especially if you're using systemd stuff, is

It's it's initially nonsensical, but once you understand the logic behind it, it makes a lot of sense. And it's kind of it's pretty straightforward in a way that I I wouldn't have imagined I would be saying six months ago. Yeah. Um so like I don't I I wouldn't be afraid of that.

If you if you have time to learn. And then f I would act especially, I would say like the modern I don't know what this philosophy is called, but modern command line stuff like Podman, System D, like they all use this kind of like command verb structure now that I'm sure you've seen, you know. Like it's it's all like system system CTL is kind of the main like system. like system D thing for interacting with the system. It's like system C CTL start, system CTL stop status, or like

Podman image pull. You know, it's it's all very like verbal now. This is the neary the neary commands are like that too. It's like yeah, if you wanna if you wanna find out about something, you use neary message. If you want to change something, you use near start neary start or near stop or whatever.

Kind of C L I like design philosophy has become much more like verbal and kind of human readable in recent years in a way that is a lot easier to get your head around. Yeah, there's a there's a lot less like, hey, you gotta do two dashes and then a capital M and then uh space and then equals and then yeah

It's it's much more human. Definitely less arcane. Uh the l last thing I'll I'll mention, um I've I've got a I've got a Pi KVM hooked up to that machine for for actual remote admin if I really need it when I'm out of the house and need to get into the BIOS of that machine or something. Um I also have a serial console. Running out of that machine. I love a serial console. Yeah.

Had to go right in there and look up the what what is it, the UART, I guess. Does it have a serial port on the motherboard or do you have to use a USB dongle? It's it's got a header. It's got the my motherboard has a a COM port header that I run to a serial connection. Wild. Good old good old D is D E nine, right? It's not D B nine. I think it's I think it's D B nine. I think it's D E nine. I don't remember. Actually. Anyway, it's one of those that I uh I run that out to the Py KVM. But it's um

And and that gives you terminal, termin uh ra a raw terminal in your Pi KVM. So Pi KVM for people who don't know is a Raspberry Pi hooked up to a capture thing. Yeah. Video capture. Yeah, video capture that lets you basically get ten ADP video uh in a remote box on the machine. There are like a ton of cheaper options for this now. Like the Pi KVM seems like the kind of exploded a market of cheap uh IP KVMs.

That are like you can get them for like a hundred bucks or less now that do basically the same thing. Well, they also have things like uh jumpers that you can plug into the power switch on the on the motherboard, stuff like that.

Uh so so yes, the w with the serial console, I have I have both video output out of the the server and serial like text output of Wild. Effectively the same thing. Can you do text input through the serial console or is it all okay yeah, so I I can SSH into the Py K Vm and then I run a screen session. That's another thing I made a little system deservice for is that GNU screen, which is like T mux, it's another one of those

terminal multiplexers, but the nice thing about screen is it can open serial connections. Uh it has built in support for that, so I just have a have a little service that starts up at boot that uh that connects to that um serial output. And the the cool thing there is that that persists reboots. Like I've got a text log of whatever the

console of that server was spitting out for as long as that screen session has been running on the Pi KVM, which in a lot of cases is like weeks or months, right? So and I can go to that buffer and like save the whole thing out to file and have like a a log of the last however many months of what the machine was doing. Like, just do a stream of that. Just put that out on Twitch. Yeah, sure. I don't I don't use that very often, but like if if that machine like crashes hard.

Yeah. It's nice to be able to go see what was going on in the serial output before it went down. It's incredible. Yeah. Yeah. It's it's it's a nice amount uh level of redundancy, but uh that that's pretty much everything. Okay, so my stuff is a little bit uh I mean it's a little more complicated in some ways and a little simpler in others. We've talked about some of it already, so I won't I won't repeat. Sorry, I didn't mean to dominate most of the episode. No server.

See, this is this is where the philosophical divide happens. You you put the work in to set it up and then when you're done you're like, all right, that's done. I'm gonna go do something else. Yeah. I do I put the same work in to get it set up and then when that's done I'm like, now what else can I fuck with? Yeah. Like I go looking for I I've got the proverbial hammer looking for a nail where you're probably like living a healthier More balanced life of other interests.

I don't know about that, but I also have other I have a e my my weekends are not really mine these days. You know, parenthood. Yeah. Um so uh I'm running the NAS, the the Synology uh It is a D S one five two O plus, which is a seller on J forty one twenty five, which is I don't know, it's it's around all it's it's Gemini Lake, I think is what the code name is for that, but it's basically four cores. It has decent quick sync, it can do four K transcodes.

Um and it has sixteen gig uh sixteen that that might have eight gigs of RAM, I can't remember. Um The so and that hold that's the bucket. Right. That's the bucket what holds the data. It has I think five five drawers. I have three drives in it usually. Oh, really? Uh yeah. The you know, uh look, man, electricity not cheap and each of those drives cost money. The thing that happened is I was looking at what drives cost today and for what I paid for those, I think, ten or twelve gig drives.

I could get like twenty gig drives now. So Yeah. Yeah. Um drives do get cheaper pretty quickly over time. What what are you running? Uh like a parity, like one parity Butter Butter F S with one parody, yeah. On the Synology Oh, I didn't know you could I thought I thought Synology you had to use like their They were

They so they used to have their own thing. They switched to Butter around the time I bought this device. This is one of the I bought this particular model because it was one of the ones that they were like, Oh yeah, you can use ButterFS on this. Oh, okay. Um So uh basically I run heavy stuff on the B Link. Uh the B Link is uh uh Intel N five one oh five which is one of the one of the low power lake chips. It's uh

um about five watts at idle uh on on that thing, which is which is nice. Jasper Lake. Jasper Lake, yeah. That's a Jasper Lake. Interesting. Uh it's for it's it's a low power for always on. Now the bad thing about that particular CPU is that it's starting to uh reach end of support in most distros. for the transcode stuff. So I'll have to either back uh grab back ports from the repos for the transcoding to work or just stay on the LTS version a little bit longer than I maybe would naturally.

Um you you know what you can do and I have run into this before if uh like if if the distro you're on stops shipping the firmware files, like the dot bin files that you need to make the quick sync device keep working, you can actually just go to the kernel tree. Like like the kernel.org tree. You can go to kernel.org and find the right dot bin firmware files for your for your for the quick sync on that thing. You can literally just copy those into the relevant directory.

And it just works. And and the kernel will just find them and load them. Or you have to I think you have to do do like a mod probe setting, you have to like tell it to load them or something. But like We can also use an older version of the kernel, right? Theoretically. But I think you're you're better off just using the stock tunnel and just telling it telling it, Hey, here's these firmware files. So you can get you can totally it's weird. The time I did that it felt wrong. I was like this

I feel like I'm messing with things I shouldn't be touching here. Realistically given the timing on the Wildcat Lake stuff and it like I'm probably just gonna not update this machine. Into an OS that doesn't support the hardware transcode on it. Oh, I'd hope that stuff comes out soon. It would be nice. Um so so yeah, my strategy is to keep every everything lives in do so on the B link, everything lives in Docker. On the Synology, everything is uh it's it's It's a little bit more of a mess.

Um, I do have a big twenty gig external drive that I sometimes plug into the Synology when I think about it to do like a backup of config and um and like the the important uh volumes in there. So the stuff that I want to make sure I don't lose. Um so wait, is the Synology running any services at all? Or is it just storage? The Synology has a couple of really light things. Like my Ubiquity console runs on the Synology because that's super light.

Um, I wanna say I have a couple other small things, but nothing like I I wouldn't run a game server on it'cause when I when I tried that it made the entire thing slow as hell. Yep. Um I don't there I have the option of putting a little SSD in there to serve as a cash and I haven't bothered because I unlike you, it's more of a dumping ground for me than a than a place that I work off of just because

Like I used to save, like I said, I used to save the the images and the video assets and stuff that I use on my OBS profiles there. And i I found that running that over gigabit that was shared with everybody else in the house made OBS really, really wobbly in a way that I didn't like. Yeah. I I've I've found over time actually like there there's some I lack the vocabulary, like there's just some sort of like file system arcana at the very low level that

just doesn't work well over a network share. Like yeah. I I finally gave up and stopped editing Audacity projects directly off of the NAS because it would just like seize up at weird random times. And when you dig into that stuff, you find again, it's like It's like such low level file system stuff that like I have no idea what I'm even reading about at that point. It's like, Oh, this doesn't support this type of indexing and blah blah blah and like

That's why this application doesn't like that. So like sometimes you have to throw in the towel and and Try not to work off of the server for literally everything. For me it was it was this is making things unpredictable and weird and annoying me and I don't like that. So I I stopped doing it. Yeah. Um I run so the Synology's ba bait bait like I said, basically just a bucket. There's a couple of really lightweight things to run there. Um I thought about actually putting a pie hole.

uh a pie hole um image on that because that would be easy to do. It's pretty light. It doesn't really take a lot of RAM or anything. And that would give me a little bit more redundancy on on that, which would be nice. That's I think that's that's worth doing. I think I think there's like ways to unify pie hole configs across multiple nodes now.

Yeah, they're just text files, so it'd be easy to move them across and I don't really touch it very much very often anymore. I just have to update the gravity every once in a while.

Um, so I added uh the B link is the is where the magic happens for me. And that's the like I said, it's the N5105. I have My strategy there is to keep everything in Docker and keep all of the data that's dynamic on the server, on the on the Synology, so that I only have to back up the Synology and like the Docker configurations for the for the different for the five or six images that are on there.

And that has worked. It has been unbelievably robust since I since I set it up, uh, probably almost four years ago at this point.

Um what are you using for the Docker configuration? Are you doing Docker Compose? Well, so you know, I've this this is a a journey of me learning how to use Docker. So in the beginning I used Docker Compose and then I switched to Docker command lines and then I installed Portainer a few years ago and just imported the existing Docker containers that are running into there and added the new ones through Portainer.

So, you know, it's a mishmash. Okay. Not the best, but it works. That works. That's fun. Um, the hard part of this was getting Docker like there's two layers of abstraction between the permissions on the Docker containers, the for that like so for example, when I'm adding my Plex files, uh me my media files to the Plex library, I have to have Uh different. Getting the permissions right inside the Docker container and inside the host for the Docker and on the Synology.

is always wonky or sure. Kind of a pain in the ass. Yeah, I can see that. And it usually involves some real bullshit like making sure the user IDs on all three machines are the same. Yeah. And when you do that it magically works and I just kind of was like, Okay, that's good. Not gonna think about this anymore. That's that's going over NFS, you said? That's going over NFS, yeah. Okay.

Um the let's see, the other thing about that is that for example, most of that access is read only. So Plex only has read only access to the media collection. Except for the one network share that's where the Plex database is stored. And that does require write access for obvious reasons, um, both for the cache and the and the updating the database.

Um and I had to I I I if I recall I think I did some real crimes to make that work, but I don't want to I probably don't want to talk about that too much. Um I also run some game servers, so I have a Linux game server host uh app there that lets you spin up like Satisfactory and Counter Strike and Quake and stuff like that.

Uh we've talked about that a little bit before. Yeah, I've used that before. It it works well. It's it's fine. It's it's not it's not as modern as like isn't there's like pterodactyl I think is one and there's another one people like as well, I forget.

So it's called I haven't tried pterodactyl since it was relatively new. Um it was a little bit heavy for that machine when I tried it. Like it felt slow. It's got like a nice elaborate web UI and stuff, right? Like Linux Linux GSM.

is pretty old school. It's basically just a bunch of like TMUX sessions and scripts, like bash scripts kind of bolting everything together, like there's nothing fancy at all there. Yeah. Um on newer games it behaves differently. Like setting up a Valheim server, setting up a satisfactory server is a little a little weirder in there.

Um and then I have a thing I use a Minecraft uh server manager because the kiddo does a fair amount of Minecrafting. And um let's see, I I can't remember what it's called and my session is timed out. So let me fix that. Um, but it it basically gives me uh overview on uh like how you log in. Like I can set up I can make a new server really easily. I can control who has access to it.

all that stuff without having to even start Minecraft. So I can do it remotely or or anywhere. Is that anywhere, which is quite nice. Is that for Java or bedrock? It's for Java. We don't fool with bedrock any bedrock is for little kids. Really? Yeah,'cause Java you can do mods, but Bedrock you can't really do mods on.

But Bedrock's the one that runs on the switch too for babies. Yeah. Uh it's called Crafty Container. Um and it it lets you you can also run Bedrock, I think, servers inside that as well now. And so you've you've really run a satisfactory server on that B link?

It was great for one player. Okay, I was yes. I've I've I've heard that the the memory scaling requirements with the size of your satisfactory world are ferocious on that thing. I think that the moment I added a second player to it, it would probably have gone straight to hell. Yeah. Um just unless you were like in exactly the same chunk of the world. Yeah. Yeah. I think I think I read they recommend like twenty four gig of RAM for like a pretty good size world.

Yeah. Or something like that. I was just like, nope. If I recall when I set that up, I set it up on my desktop machine and then I migrated the configuration over to the little server. Yeah. Um and I didn't I didn't play it for super duper long. It was it was Like I said, Val Valheim ran great. Valheim was really good. I remember small people on different times. Man, that's not bad. I Valheim I remember being pretty modest, like two to four gigabytes.

total or something like that? It was light. I gave it I gave it eight gigs of RAM, at like half the RAM on the little server and it was fine. Um, I bet that if you had hit Plex at the same time, you would have noticed though. Sure. Um, and and so the way those Minecraft servers are set up is that.

When I wanna move them to the latest version when the kiddo is ready to switch to a new, you know, because they update Minecraft a couple of times a you know, through pretty much once a month, it seems like. All I have to do is restart the portainer and it just downloads the latest binary and then if I want to use one of the older ones I can go in and manually change it.

Otherwise it's pretty good. And then I also have uh Jellyfin and some other stuff running there that I don't I don't actually run Jellyfin all the time'cause it's it's also it was too heavy for this machine, I think, the server The experience I had running the server on my desktop PC with sixteen cores and sixty four gigs of RAM and the experience I had running the Jellyfin server on the sixteen gigabyte

Four core machine were wildly different and pretty bad. So how long has it been since you ran Jellyfan? Uh mid midway through last year probably. Oh gosh, I'm trying to remember when it was. They they put out an update, I think it was later last year. They they they basically did one of those classic open source or open source moves of Hey, this project is pretty old and has a huge amount of tech debt.

Oh. And we have not had enough contributors to deal with it until now. Like we're finally biting the bullet and like re architecting a ton of the Oh. Under the hood stuff that's been holding this project back. So they had their huge release and it was like serious. What did we talk about it on here? No. Like the the upgrade. Like the upgrade notes were crazy. They were like

They were like, make sure you back up your database before this because if the upgrade fails, it will hose your library. That's bad. Um Depending on the complexity of your library, this could take several hours. So you might want to start the upgrade before you go to like before you go to bed overnight and type stuff. Mine took like thirty seconds'cause again it didn't have to be. Well and yeah, and I just don't have a gigantic library. This was like

This was for people who had a lot of stuff in there. Well so I have ten or fifteen terabytes of of D V D's and Blu rays. That's a good that's a good I y yes, I've seen your plex. You've got a lot of stuff. That might be a but but the point is I don't I can't say for sure that those upgrades they're they are making help with the system requirements necessarily, but they might. I mean, the problem I had honestly was that the initial media scan was going to take like four days.

And uh pulling all the hefty. Yeah, pulling all the d the metadata and stuff down was Bad and slow. full database redesign, I believe, is what they did effectively. Like they modernized the data database they claim. So it maybe not right away, but a couple more versions from now it might be worth taking another look at. Yeah, I'll I'll take a look. I I I like it. I I mean I paid for a lifetime Blackspass a decade ago. So if

If I feel okay. If I if I had a Plex Pass, I probably would not use Jellyfin as much, but um and then I have a Perforce server running because you can run a five seat Perforce server without paying them anything. And um I use it to keep s like small projects I'm working on on there and it's it hooks into the NAS so it has redundant data storage and like I said, it's it's like everything else. It's running in a Docker container and then then the data's actually stored on the on the machine.

Um on the on the NAS. Um so then uh in addition to that, I have like the home assistant yellow, which we've done we did an episode about last year. I'm not gonna get super into that, but that lives in a closet in the center of my house, um po over powered over powered over Ethernet, which remains in twenty twenty six magical to me. Um, that controls all the home automation stuff, all the lights and all that. And if if you're looking for the thing that I spent time futzing with.

It's probably that because I can sit on my laptop in the living room and do it really easily. Sure. Um I also have the pie hole. I'm down to the one because of the stuff we talked about a couple of weeks ago on one of the episodes. I the the the catch up episode, I think.

Um, and then I have some upcoming stuff that I want to get into. Um, I don't know that I'm gonna really fool with adding anything new to that existing B Link machine right now, just'cause I it's it's on borrowed time. Um when the and and when the w Wildcat Lake machines come out. Mm-hmm. Um but I do want to have snapper running on that just so I have

uh kind of upgrade insurance. It's always a little hinky. Like the LTS, the the scary thing about the LTS distros is that they update so infrequently that each one feels like it's rife with peril. Like like you know, I read the notes, I do what they say. But I feel like It's been long enough since the last time I did something, I set this machine up that if it if something breaks, it's gonna be a real pain in the ass to reverse engineer.

I I actually I actually hosed my Debian bookworm install on that server to Trixie and ended up just doing a wiping and doing a fresh Trixie install, but I had a snapshot of First of all, it wasn't the upgrade pr it was it was their upgrade instructions that I fucked up. Like I ran one of their like, oh you should like clean unused, like obsolete packages or something like that. And I ran their I ran their recommended arcane apt command.

Mm-hmm. And it broke the machine's ability to boot. Yep. That'll happen. Anyway, I was able to restore config from the snapshot pretty easily, but you're th the point is like OS upgrades, distro upgrades are not Without risk. Yeah, they're scary. Um, the other thing is I want to add a reverse proxy so that I because right now all of the subsurfices that run in Docker on that B-link are on Port one five three nine or one five three seven or one five four eight.

Yeah, and it's all the same IP and it's it's it's fairly frustrating. Yeah, I I get it, I know what you mean. But also, I mean I can tell you right now that Jellyfin's web port is eighty ninety six and own tones is thirty six eighty nine because I I once thought as you did

I ran I tried to I tried to run Nginx as a reverse proxy for a little while to solve this exact problem. Yeah. And it went great, right? And it was awesome. You do it every day. It was it was so difficult to get things working right that I just gave up on it and went back to using port numbers for everything. So then maybe I need to post it with all the port numbers I stick on the side of my monitor. So so that said Nginx is like the manual.

difficult way to do it. Oh, you you did it the hard way? I'm shocked. Yes, yes. Um traffic is another one. You're familiar with traffic, right? T R A Right Yeah. T R A E F I K is another uh reverse proxy that people on the Discord really like, and I believe that one handles a lot of these specific proxy configuration much more seamlessly for you. Okay. I'll give that a try. Is is the one, especially if you're using Docker. Um

I want the one for babies. I want the one that a baby can do. So if if you're running traffic and Docker on the same machine, traffic can actually just plug straight into your Docker configuration and like kind of set up proxy stuff without you even having to do anything? Oh.

But I d I d I haven't messed with that. I I would be running it on a separate Raspberry Pi, so I don't know how th if that would've helped me or not. But point is that there are other and caddy is another one. I since you're if you're just writing down I'm just taking notes. Two Ds or one D? Two two D, C A D D Y, Caddy. Caddy and traffic are both apparently quite a bit easier than Nginx to configure. It wasn't the reverse proxying that I had a problem with. That worked fine.

The problem is that most modern web pages and like web interfaces for these services are way more complex than just proxying a single URL to another computer or another address. Yeah. There's all there's all this like WebSocket business going on that I barely understand.

So like every service is slightly different in the configuration you need to like get it to pass everything through to like like I would get the like the the page would load through the proxy just fine, but like say Top top UI bar was missing, you know, or like Yeah. So so and so controls were not working or didn't show up. Hey, there's a thing in my screenshots of this on the website that shows you how to do it that just doesn't exist on mine. It was it was yes, it was just

like the they basically proxied but were kind of unusable. It was it was a pain. Well and that and that's pretty much it for me though. That's like that's what I got. Um I am excited about replacing that little B link so I can run some heftier stuff. I kind of wish I like I wish that there was a six or eight core low power equivalent. Like like I keep looking at those minis forum boards that are like mobile processors from laptops. So they they do the thing where they solder

mobile processors onto like PC motherboards, like ITX motherboards or or whatever. Nob nobody ever did that with LunarLate, did they? Uh Lunar Lake is kind of hard to get. You can get those on Lunaric, but they're pretty expensive. Yeah, I mean that platform's kind of morbund anyway, right? Yeah, and I think honestly the where the thirty two gigs is fine on my laptop, I think I would want

I dream of a world where RAM is inexpensive again. I can sure more RAM than I need in a machine. Um, so yeah, I I would yeah, maybe Phantom Lake though. Panther Lake. Panther Lake, yes. The the last leaks about Wildcat Lake I saw are there's like a s kind of a second gen refresh that are is already showing up in these CPU.

Uh in the leakers. Yeah. Whatever whatever these like CPU testing databases are on the internet where like new SKUs of CPU just show up randomly. The point is there are there are CPUs or SKUs at that tier that have way more cores is the word. Oh. Like kind of exactly what you're talking about. Basically at the Wildcat Lake tier, but there's one with like

I forget what it was, like six P cores instead of two or something like that. Well they they showed one with six um with two P cores and four L P E cores. That I think we talked about the other day at CES. The two and four is the sort of classic configuration that has been yeah has been known uh for a while. Hang on, I'm I'm actually clicking through right now. I can just tell you. Oh like four four P cores and four LPE cores, it looks like is maybe what the refresh design is.

looking like. Anyway. Yeah, I'd take that. I I would I would love to see them broaden that product line a little bit.'Cause these these little these little mini PC type boxes are rad and do all kinds of cool stuff now. And they are like it it The Pi ho the Raspberry Pis are neat and when you talk about like a Pi five or something, you're talking about actually a real capable computer, but they are ARM and then you have to deal with ARM packages, which like is doable but

Not everything is compiled for ARM and I don't wanna have to compile everything that I run, especially when I'm running it on a pie. I don't wanna I especially don't wanna have to compile it on the Pi. So I mean the the bigger thing for me at this point is that the Pies just keep getting more expensive. Yes. At at the price the price the price range you're getting into now, like you're

It's getting to the point where like, ah, one of these X eighty six boxes would just make more sense. I'm I'm getting way more expandability and power for not all that much more money. Like I I think I found that B Link for like a hundred and twenty five bucks on sale at some point. Yeah. And and that's that's a pretty like that's that's Pi five territory. Like a like a fully kitted out Pi five is probably gonna cost you more than that at this point.

So anyway, yeah. So that's it. I hope uh hopefully you all enjoyed uh the home lab uh kind of check in where we're at. Um, w I'd love to know what you all are running and what you think we're doing right. If there's stuff you think we shouldn't be doing or stuff you think that we would benefit from knowing about, please post it in the Discord. And if you aren't or send an email to techbot at content.down.

And if you aren't in the Discord, you can get there by subscribing to the Patreon. We're a listener supported show, which means we're only here Uh, because of you, the the the listeners. Very sure. So uh you can go to patreon.com slash tech pod where for as little as five dollars a month. Uh that's like I don't know about you, but at the Starbucks by my house now, that's less than one cup of coffee. Inflation has has uh come to coffee in Pacifica. Hey, their prices are going up, ours are not.

Yeah, that's true. That's true. Um, yeah, you can get access to the Discord, you get access to the monthly Patreon exclusive episodes. We just kinda chat about what's going on and and uh sometimes topics that are too small for the regular show. Uh and uh the Discord is full of bright and and uh uh clever people doing weird projects and fun projects and stuff like that. I just got a package.

that has been talked about uh for for a project that has been talked about by multiple people in the Discord over the last couple of years. I'm not gonna say what it is. I'm just gonna leave it hanging. Oh, I really wanna know. I know. Well there's only one way to know. You have to go to patreon dot com slash checkbutt. Well you don't have to, but everybody else does.

Uh we'll talk about it in an upcoming episode. But uh thanks everybody for supporting us. As always, thank you, a very special thank you to our executive producer, to your patrons, including Jason Lee. And we Felicitas Rips, Andrew Slosky, Jordan Lippett, Bunny Zero, David Allen, James Kamick, and Pantheon, makers of the HS three high speed three D printer. You know

Well I need a 3D printer for this upcoming project, so I might have to reach out to Pantheon makers of the HS3 high speed three D printer and see if they can print something for me. I think you you might have an avenue for doing so. Yeah, if only I knew.

Uh thanks everybody for supporting the show. We do really appreciate you and uh hope you have a lovely, lovely week. We will be back next week with another edition of the Tech Pod. I'll see you then, Brad. And always, as always, please consider the environment before you print this podcast.