322: It Was DNS

So last week after we recorded the podcast, I had my phone up all the way because, you know, we do the beeps to sync up the episode. And then I went out after the podcast and I was looking at the Internet and it started playing a video.

and it was a video of animals making weird noises and uh it scared the hell out of everybody in the living room with me because it was it was like loud duck eating watermelon and and peas noises like i'm not gonna make the noise but it was just you should just google ducks eating watermelon it's pretty good way ahead of you buddy but you watching the watermelon ducks maybe maybe look

I love it. I love a watermelon duck. I'm not here to divulge my extracurricular activities, but. Look, chinchillas doing dust baths, always good. I've never understood that. Owning guinea pigs got us adjacent enough to chinchilla care that I came across like.

chinchilla facts on the regular oh yeah no it's a commitment and the dust bath thing like it sounds kind of cool in a way but i don't know I there's a there's a woman that we follow that my daughter and I watch sometimes that has chinchillas and like she's she's a she's a good chinchilla influencer because her thing is Yo, nobody should get chinchillas because you have to keep your house at 60 degrees or they die. Oh, what? Because they're from the Andes. So it's high and cold. Yeah, sure.

And like she has to wear a respirator. She wears a respirator and has a broom in her house. That's basically like, here's where the here's where the chinchillas get the dust baths. That's intense. Guinea pigs are also from the Andes, but maybe maybe like the lower Andes because it's not quite that. Restrictive? Anyway. Yeah. I mean, look, they're all rodents in arms. That's just how it is. They are going to rise up one day.

Okay, so I have a confession, I guess. I started the call today by saying I discovered I had 277 tabs open in Firefox. That was too many tabs. I think you can do better than that. You know what that sounds like to me? A good start. It took me an hour to trim it down. Now I'm only like two pages on my scroll. I mean, what is that? That's save tabs, I assume. You're not hibernating and resuming.

every day with a running ephemeral browser session, right? Like those are actually saved and will restart on another machine and so forth. Now, see, it's interesting you say that. It is an eternal session. I never clear my Firefox session. Wait, this is literally one process that's been running for? Well, no. I mean, I turn off the computer every night. Just restore the session. That's what I mean. They're saved half. That's what I'm saying.

I mean, what I mean is like literally the same process ID running for days or weeks at a time. Like you start a browser and it's that process running and doesn't get closed and just gets shut down. I mean, not shut down, but. hibernated and resumed like i said look man i'm paying 78 kilowatts cents per kilowatt hour here i can't afford to leave a computer running 24 7 are you kidding you can sleep it you can hibernate it

You know, it's still, I'm not going to, it's not worth 10 watts to, you know, but the tabs restore when I start. It just comes right back. That's what I meant about these are, this is a session being restored, not one browser running eternally, which is.

kind of horrifying when you think about it yeah i thought about i mean because you can do those container you can do like the tab groups now in in pretty much all major browsers i think where i could save these and move them from computer to computer but you know the tabs are these days the tabs are machine specific it's like hey here's how do i get my nvidia driver pass through the kernel or whatever on on on this one and and so on and so forth but i i just realized

On the Windows PC, eventually Firefox would start to get kind of chuggy if I had too many tabs open. and on the linux machine i never and that's usually my sign to like you know trim the fat a little bit and on this one it hasn't been a problem so i've just been like i have tabs that are a month and a half old over here and uh

Anyway, I've got to come up with a new strategy, I guess. How many windows are we talking? One. What? What? How many windows do you have open? Does Firefox have some facility for showing tabs you've got open to other? Like Chrome does the same thing. I love that. The tab bar where the tabs live is no longer sufficient for the way that people use browser tabs. And now we need a second facility for showing you the tabs you have open besides the tabs you have open.

Look, there are a lot of tab crimes happening, but I am not a monster with like 15 browser windows open. You can't do that because then it doesn't restore right. Right. You close the wrong window and you and your whole thing gets out of whack. And then you lose the window that has the pin tabs, which realistically.

is the important one. I've got, let me see, I've got one, two, I've got like two, I've got two browser windows open right now. Madness. I've got maybe some total of like eight tabs open on my machine at this moment.

so the problem with firefox i mean actually there's a lot of problems with firefox we haven't talked about this yet but they're ai-ing the crap out of firefox and it's kind of jacking it up i gotta find a new browser yep good luck with that um but but the other thing is So they have the view recent browsing across windows and devices. That's one that shows you the synced tabs from other devices, which is really useful, as well as like recently closed tabs and currently open tabs.

And that's how I got the list with the 277 tabs in it. The other thing is they have this side panel that pops out where there's a bunch of AI crap I've turned off, but that also has tabs from other devices. And then it also has history and it has bookmarks. And then there's a dropdown on the far right side of the top of the window that just has the scrollable list of all the tabs. But realistically...

When you type anything in the location bar it also searches the tabs now So if I have something that's close if I need to find a tab I just type for the thing I'm looking for it pops up or I just open a new tab because Tabs is free, baby. If you are searching and it's opening a tab that was open because you searched. Yeah. What was the point of leaving that tab open in the first place?

Well, because it wouldn't have, the tab search wouldn't have come up. It doesn't search history. It only searches open tabs. It doesn't search history. Oh, this is the say tab specific search. This is not a global. Well, I thought Firefox was the, Firefox was the innovator. If I remember, didn't they call it?

This is a terrible, this is such a like mid 2000s name, but didn't they call it the awesome bar for a while? I'm pretty sure. That seems pretty likely. I'm pretty sure that name, but I'm pretty sure Firefox was the innovator of like, hey, the address bar is now going to search like.

history and bookmarks and like everything you've ever done on this browser is all going to get rolled up into what you type into this bar and it's just going to find your thing so yeah i just searched for something that like i was i tab i just closed and like so i searched for kernel

And I get, yeah, I get, I get some searches I've done in the past and then I don't get the past history. So they must not search through the history anymore. That's a bummer. My guess is that got too onerous. I guess like a Chrome still does that. Like that's kind of the.

way to use a browser to me i don't know man i like i like searching through my tabs i like having a bunch of tabs i'm not gonna let you shame me all right i'm gonna have 277 tabs brad you can't you're not my dad you can't tell me what to do look not everybody has the spare ram abused their browser in such a way, but look, I'm not going to tell you what to do, but I've been on record many times. Cold boot, fresh browser every morning is clean living. Do you not even open your pins?

Not what? I don't even have pins. Do you not do pins? Oh, not unless I need them. Oh, my God. I have pins for my mail clients and the home assistant and the Twitch panel. Of course, I open Slack and email every morning. You don't use the client for Slack? Oh. Oh boy.

Welcome to Brad and Will Made a Tech Pod. I'm Will. I am Brad. Hello. Hi. I said I am Brad that time. I thought I would... Brad I am. Uh-huh. That's me.

We should soothe this up one day. No contractions on this podcast. I am here to talk in a more formal manner this week. You know, it's funny. I pointed that out to my daughter when we were watching the Stranger Things finale because... the kids that were like you know the the human experimentation children grew up in a fixed environment don't use contractions in the same way that like mr data and i think did spock not use contractions either

Or was he theoretically didn't use them very often because they messed up sometimes, I bet. I have now seen two of three seasons of the original series, but I can't. Okay, I'm just kind of trying to run through my head. I bet he did use contractions, actually, if I had to guess.

But Data, famously, not a contraction user. Are you telling me you don't have an open tab already with the search? Did Spock use contractions? I was typing it, but I was trying to go quietly. In 277 tabs, you must have... Like if you open enough browser tabs, then every possible permutation of every search that has ever existed will be in there eventually. Right. Okay. So I went to the Trek BBS, the number one result for this. Oh yeah. And it's a, it's a, it's a board.

This is an old forum. User Tellerites says, I don't, or should I say, I do not think that he did. Did Spock ever speak with a contraction is the subject? I don't, or should I say, do not think he did, but I've always been curious if some slip by. If so, which episodes? And then 1001001, who has a Calculon avatar, said he used several of them in Star Trek IV, the voyage home. What is a Calculon? Calculon is the actor robot from Futurama. He's the Shatner. He's Futurama Shatner.

And also then a guy with a minion avatar says it was his second line in the cage was a contraction. So Spock, no data definitive. No, he's not. If we're ranking Star Trek characters, I'm going to put. data above Mr. Spock. Hang on. I know this. We're already planning for this coming holiday. I've got this holiday's ranking right here in front of me. Ranking of Spocks. Ranking what? Ranking of Spocks. Yeah. Oh. There's only four, aren't there?

No, there's three. Okay, first we would have to define what constitutes a Spock. Is Data a Spock? Absolutely. What? No, he's a robot. He's not a Spock. No, 100% he's a Spock. We hashed this whole thing out on the WatchCast already. Oh, no. Every Star Trek has a Spock. Data is the Spock of Next Generation. Odo is the Spock of DS9. Is Ensign Ro a Spock? No, you only have one Spock. I mean, I don't know. I think Spock's, I think some people would disagree with that.

Sometimes you have two Spocks. Mirror Universe Spock is also a Spock. Or is he not a Spock? Okay. I'm trying to think this through here. There's some... All datas are Spocks, but not all Spocks are... Let's move on. With great Spock comes great... data responsibility responsibility yeah um we're doing a we're doing a potpourri this week we had some weird stuff some stuff's been going on we got stuff to talk about i think i think we should start

with moonlight on the xbox because okay i'm going it's unprecedented it's the last thing on the list but i think we should start there because it's fun Yes. I'm going to let you talk while I, I failed to do my due diligence, but, but somebody on the discord in particular brought this to our attention and I've got to credit that person because that was talking about Hobbs. Was it Hobbs?

Hobbs gave me the way that I'm now changing so we talked about a couple weeks ago Apollo which is a sunshine fork that okay let's take a step back Sunshine and Moonlight are the server and client respectively that replicate the old NVIDIA GameStream functionality from the GeForce experience. which was set up originally to let you stream games from your GeForce equipped PC to an Nvidia Shield. Now, somewhat of a niche category.

But the protocol was really good and people made, when they pulled it from the GeForce Experience app, folks basically created an open source server called Sunshine and an open source client called Moonlight. that would work on kind of everything, like Windows, Mac, Apple TV, iOS, Linux, Xbox. a bunch of TVs, like smart TVs that you can install your own apps on, all sorts of stuff. And the upshot is you have a really clean, pretty high quality way to

transcode the video with very low latency on your desktop PC, and then stream it to any other screen in your house. Yes, particularly if you run a hard line all the way down the hall to your living room, Adam Patrick Murray. Yeah, that is one important. I was thinking about that. I call that the Adam principle. I've been sitting on that call out for a month now.

oh you can also do it like just to be clear it also works really well over wi-fi i use it on wi-fi in the stream deck all the time yeah i believe it i mean assuming well you know assuming your wi-fi is clean enough and you're not assuming you're on like Let's say five or six, five, six or newer Wi-Fi standards would be my guess. Anyway.

Okay, so folks have been talking about this Moonlight Client for Xbox because I've talked about using the Steam Deck one in the past and using it docked with the TV, but it's a little fiddly. It's fine, but the Steam Deck is... clearly built for streaming 1280 by 800 or playing stuff at 1280 by 800 on that device and it's a little hinky when you plug it into an htmi port yeah and and i should mention i think it

Was probably on this very podcast that I talked recently about wanting to get the new Apple TV, which is still not out yet. Because I was thinking about using that as a, as a moonlight. Or even Steam. I thought about also trying the official built-in Steam client, or I guess they still call it Steam Link. Steam Link, yeah. Is what the app is called. But either way, I thought about using an Apple TV as a streaming device. And then...

I looked, I could not find the user. It might've been somebody on the next land or discord actually that mentioned it to me first. I forget, but whoever it was, thank you. That mentioned there is a moonlight client on the Xbox. On the App Store, yeah. It's worth mentioning the Apple TV one has input latency problems. I tried it again. The Apple TV I have is old enough that it wasn't great.

But it's possible. I'm, I'm, I think one generation back from the current one that you can buy today, which will be two generations back from the new one. Yeah. Some, some people on our discord, I think tried it on the newest one and said they didn't.

minded or they thought it felt fine i mean latency is something that's always going to be subjective based on well personal sensitivity some people are going to notice it more than others but the bigger thing with the apple tv is it's not a gaming box you know like yeah the current models Now, and I assume this next model probably as well are capped at 60 hertz, for example, so you can't do higher frame rates. You know, it probably does support 4K and HDR just fine, I would assume, but it's not.

It's a streaming box for video. It's not a gaming device first. But you know what is a gaming device first? Yeah, the Xbox. Or the Xbox Series X and S. And sure enough, there's a UWP Moonlight app on there. I don't know if UWP is even still in vogue or not. Yeah, it's one of the ways you can ship games on the Windows Store as well. Yeah, it's one of those cross-platform Microsoft frameworks that lets you kind of make one app that works on multiple devices.

Yeah, it used to be the only way to ship a game on the game pass game on Windows was it had to be a UWP app for a long time. I think they've changed that since then. But. The upshot is we've both been, I think we both used it. Have you used it at this point or no? I have set it up. The streaming is working fine. I mean, maybe I'll let you go first because you've had more luck with it. I also just haven't had a ton of time to fiddle with the...

nuances of the config. It's streaming to my TV fine, to the Xbox fine, but it's still opening things on my PC, on my desktop. When you launch the... It's still like when I launch Steam. like the steam windows still opening on my PC monitor, not on, like I basically get a blank desktop streaming to my, uh, Xbox, but none of them, none of the windows are actually appearing on it. They're all there. It looks like there are multiple ways to deal with that. You have to like,

fiddle with the windows sort of, I started to say preferences, but I don't mean preferences in the sense of a dialogue where you are tweaking settings. I mean like windows is preference for which. monitor to output to, you know, like Windows' opinion of which one it thinks is the primary display. So on Windows, it's always just streamed my primary monitor, which hasn't...

been a problem. I think that might be a setting. I'm going to be real. I don't love the sunshine and Apollo because Apollo is a fork. So it's basically the same thing in this regard. I don't love the. Web based settings for sunshine. Like I find them just maybe a little bit hard to parse. They're not overly explanatory. I think maybe a little bit more like flavor or help text would help.

Well, and the weird thing is they split settings into two different sections. There's like the server settings and then there's also per application settings. So like when you connect to a Moonlight, sorry, when you connect to a Sunshine server with your Moonlight client, it's...

It's like, hey, which of these things do you wanna launch? Do you wanna launch the desktop? Do you wanna launch Steam big picture? Do you wanna go straight into a specific game? And the right answer is different for kind of each.

Like there's no universal right answer, unfortunately. Yeah, it seems like quite a bit of fiddling you might need to do per both client and server to get everything working right. Anyway, like I was able to pull up the connection stats on the Xbox and see that it was streaming 4K 120.

HDR just fine. Oh, that's nice. It's just that I just need to put the work in to try to, and it may be something like disabling my local monitors on the PC, whatever I'm streaming, because I think it can do that too. I think Apollo can handle that. So Apollo does that on Windows seamlessly. I found Apollo to be basically zero fiddling and just installed it and it was running and it was right. I might have had to specify the primary monitor, but I think it just pulled the primary and.

I have Steam Big Picture set to launch on the primary monitor. So when it launches Steam Big Picture and it starts streaming that to the TV, it just worked. There was no change. Okay, but to be clear, the reason I... even need to worry about this is because my monitors are lower res than my TV. I have 1440p monitors and a 4K TV, so I don't want it streaming 1440p to the TV. Do you have a dongle?

So that's one of the things Apollo does is it just creates a dummy virtual monitor. That is the big thing. Sorry, I was looking for that Discord stuff while you were giving the top line here. I'm not sure if you mentioned the Apollo difference.

Yeah, the Apollo difference is that it switches the resolution to your output resolution and turns off your other monitors if you tell it to. Yeah, so I'm not telling it to turn off the monitors, but yes, it detects the resolution and frame rate and color depth and everything of the target client that is requesting to connect.

And it just creates a virtual desktop at those settings and streams that. So apparently you can get sunshine to do that, but it's work. So sunshine. So, okay. So the TLDR on this is that if you want to do this and you're curious about it, um,

I would recommend Apollo on the Windows side. And if you're running Linux, then Sunshine is probably your option. Apollo was kind of hinky on the Linux side. Yeah, I think Apollo from the sound things is probably using some pretty like Windows centric stuff under the hood. It is the virtual monitor thing doesn't work on Linux for Linux related reasons, but also Linux provides most Linux distros and desktop environments and window managers provide.

Command line tools that you can easily use to turn on and off monitors at will or displays at will so But you will for sunshine to do a virtual monitor You will either need an HDMI dongle or like in my case. I just have the HDMI goes out to a capture card on my streaming PC and I set that machine to always provide USB power to even when it's turned off.

and that capture card stays hot and that shows up as an option so I can turn it on and off at will. Sure. Are there not other ways to define virtual desktops? Like I know the NVIDIA drivers on Windows, you can just create a virtual...

or sorry, virtual is not the right term, but kind of a, you can create resolutions that exceed your physical desktop resolution. Like I've done that before. You can, you can even on like a 1080p monitor, you can define a 4k resolution for example and then just have your graphics card scale that down to 1080 so yeah uh on linux you can if you're using x11 and if you're using wayland you can't is the current current state of the art as i understand it okay um

Anyway, the upshot is on Linux, on Windows, Apollo handles a lot of this stuff for you. It's really nice. I was able to do HDR streaming to the Xbox Series S in the living room. I don't have an HDR TV in there, so that looked really bad. But when I turned it off and put it on SDR, it was fine. Here's a question. Yes. Hi. Were you able to tell Apollo or you were using Sunshine, not Apollo, right? Either way.

I was using Apollo on Windows. I've used Sunshine on Linux. I'm using both right now, depending on which machine I'm booted into. Are you able to not have HDR enabled on your PC, but then have it turn HDR on just when it's streaming to your moonlight? client because that would be ideal for me because the monitors I have in the office here are terrible for HDR. So I don't want to use HDR on my desktop, but it would be nice for that to be able to flip HDR on.

automatically when it starts streaming to my TV, which is a good HDR display? So I think with Apollo and the virtual monitor thing, you can do that. Okay. With Sunshine, I've never... I've never successfully gotten that to work. If I want to stream HDR to my Steam Deck or something, which has an HDR display, I have to turn HDR on on Windows first and then start Sunshine, and then it works. Okay.

But anyway, I was going to say, we talked about Hobbs in one of the chats about this on the Discord had pointed me to... To replicate that Apollo functionality in Sunshine on Linux had pointing me to the actions. that are tied to an application. So in Sunshine, this is the Applications tab, and it lists the things that you see when you connect to the server, right? So it's like desktop, low-res desktop, and big picture. And in that...

In that setting, there are commands and undo commands that you can set to run before the application launches and after the application is done, after you disconnect. And so I just put Neary message output commands in there that turn off my DisplayPort monitors and turn on my HDMI monitor and set the HDMI resolution to the resolution that I'm streaming at most regularly.

And it's worked fabulously It means that I hit it I see the desktop for like a flash of a second and then I get the same big picture launch and Everything's groovy. So Highly recommend. Yeah, I've got to put more work into this. Maybe I'll update on this month's patron episode, I think. Hopefully by then I will have time to wrangle. It's again, like I said, the settings I find to be a bit overwrought or just kind of.

mess to dig through um for the sunshine server and and also dealing with like when the windows like which monitor to output settings are just kind of a pain and i just have not spent the time so the thing i will say is that generally speaking especially if you have an nvidia card just saying hey use the nv anc encoder and um setting the display

and then letting it figure out the bit rate and the targets especially if you're on ethernet is fine um you can you can give it a ridiculous bit rate like 50 megabits on on gigabit and it'll just it'll work itself out I mean, why not 100? I'm doing 1080p most of the time. So and I also connect with Wi-Fi sometimes. So I keep it reasonable. That's fair. That's fair. But also you can set it to just let the client specify the bit rate that it wants. Yeah.

which is probably the better solution for my particular use case where i'm sometimes streaming at ap sometimes streaming it at 800B and sometimes streaming at 4K to a laptop. I wonder if there are any differences between like VBR and CBR situations there. Constant bit rate versus variable bit rate. Like is there overhead as it's trying to figure out?

you know, as it's adapting its bitrate dynamically to whatever the demands of the stream are? Or like, is it more efficient to just set if you've got the bandwidth over the network and a good encoder? Is it more efficient to just set a high ceiling CBR and just let it go, I wonder? So I, the one thing I tweaked was I turned off the two pass mode that does a quarter resolution scan pass just to reduce a little bit of latency.

Yeah. So like I have both latency and also just like maybe some lag in adapting to changes in the content of the video, you know, like you might get some brief macro blocking before it catches up and ups the bitrate or something. Like if you've got, if you've got the encoder and bandwidth, I wonder if it's better to just.

Just set a high number and let it go. The thing that I've found, having used this quite a bit over the last couple of years, is that I typically, I find I only play games that I'm playing with a gamepad. I don't play a ton of stuff like I don't play competitive games ever in the living room. It would be the same for, well, I mean, I might play a somewhat competitive game on a controller maybe, but it would definitely be a lot of like lean back on the couch with a controller type stuff.

Yeah, it's a lot of RPGs and stuff like that. The closest thing to a timing game I played was probably Ghost of Tsushima, which was fine. Like I was able to do parries and blocks and all that stuff without any problem. And I don't know that I would play like Sekiro or something like that on streamed.

Yeah, I think. But maybe, I don't know. I'm bad at Sekiro anyway, so who knows? I think the first thing I might play when I get this working is Routine, that horror, sci-fi horror game that came out last month. I haven't seen a lot of people talking about it that much, though.

It looks good. I played Dispatch all the way through the other day. We were sitting in the living room and making decisions together. The controller, I think this is that game specific thing. It seems like it's hard to do the Dispatch portion of that.

with the controller and score well. Yeah. But that might, maybe I just suck at dispatching. I don't know. Yeah. I am. I am going to finalize this stuff soon though, because it's not that I don't want to play PC games. I just don't want to sit at the PC to play them. When I'm already spending 10 hours a day in front of this computer anyway. It's nice to be able to play something that's like a big, giant, beefy game with my nice computer, but sit in the living room and like...

have the big TV and the speakers and the whole deal. It's time. Um, now I will say I've talked to some people on both discords, next lander and tech pod about this. And it seems like that Xbox moonlight client is known to be a bit crashy.

I have not had any crashes with it. I've had it crash once just in the time I was preparing and fiddling with settings a little bit. So that is maybe something to be aware of. The other thing, which I think I might have mentioned on the patron episode when we briefly talked about this last month. is the divergence between Apollo and Sunshine, according to the author of Apollo, is going to widen at some point.

Yeah, but he has to build clients for the other stuff before he does that. So, well, yeah, but I don't, the odds of the odds. So basically he forked Apollo from sunshine. If you want to see some real good classic open source drama. Go to the Apollo repo and scroll to the bottom and read the three paragraphs where he talks about trying to participate in the Sunshine development process and being rebuffed and then banned from their discord. And he seems very upset about it.

Wow. Which is what led to the forking of Apollo. Anyway, the point being, he said explicitly, eventually I will break compatibility with existing Moonlight clients in favor of clients that are specifically made for Apollo.

Not maliciously, it sounds like more because he's going to work on new features that will require those clients. But that's worth keeping in mind before you get super invested in an Apollo workflow, particularly in the case of the Xbox where... I feel like the odds of an Artemis client getting made for the Xbox, considering the general state of the Xbox platform, I feel are quite low.

Look, I've turned my Xbox on more in the last two weeks than I have in the two previous years. Yeah, you've been turning your Xbox on every day to play games on somebody else's platform. Look. On the other hand, that's part of their whole business model forthcoming. Yeah, everything's an Xbox, Brad. That's where things are going for them, so maybe that's fine. I will say, okay, first off, I'm going to say getting banned from something feels bad all the time.

I got banned from Reddit recently for things that I didn't do. If anybody at Reddit, if anybody's listening here, works at Reddit and wants to help me get my 10-year-old account unbanned, that would be dope. But at some point, Reddit let a bazillion people sign up for accounts using my email address. Even though they were all unconfirmed, I got hit for spam. Sure.

So anyway, I've got the disclaimer in front of me right here. I got kicked from Moonlight and Sunshine's Discord server and banned from Sunshine's GitHub repo literally for helping people out is how this starts. Oh, boy. So you can kind of guess where that's probably going. I feel this, brother.

This is what I got for finding a bug, opened an issue, getting no response, troubleshoot myself, fixed the issue myself, shared it by PR to the main repo, hoping my efforts can help someone else during the maintenance. He goes on. You call somebody a name or something? No, I have no idea. Zero idea. Scene drama. Yes. As I like to say often, open source is an exercise in self-governance. That's true.

And, you know, like sometimes it's a little messy. I mean, look, self-governance is messy. We've all learned that. Yes, that's for sure. So anyway, yeah, that's Moonlight and Sunshine and Apollo. Like two thumbs up to the whole stack. It's real good. Yeah, that's cool. Like I said, possibly more to come as I get mine working on the patron episode. I just want to say also as an anecdote last year, maybe the year before now.

I was doing some testing for this at PC World, and I had left my computer running that day, apparently. And when I connected, I connected with Moonlight at the PC World office. to my house 10, 15 miles away, and I thought I was connecting, I wasn't paying attention to which one I connected to, because I had both settings on my Steam Deck, and I thought I was connecting to the one in the other side of the office.

and i was connecting to the one across the whole wide internet and i couldn't from from a performance standpoint couldn't tell the difference so if that's not a proof of concept what is yeah it's pretty good You have to open. I think I had the reverse proxy set up for that. But anyway. Yeah. All right. Self-governance is messy. Yeah. You know what else is messy? Electricity. Electricity. Yeah.

Uh, I've got a whole stack of UPS weirdness that I want to step through here just to get your opinion on things. Yeah. Cause I'm sort of at a loss, like maybe more of a FedEx guy personally. Great. Okay. I think you, We talked about this also last, so as of last month's patron episode, I think I had ordered new batteries for my UPS and they had not arrived.

You were excited about him. You seemed cautiously optimistic, let's say. Some events have transpired since then, but I think you mentioned on that episode, you said you used to be a UPS guy and have now stopped. Is that correct? When I bought this house, I paid an electrician to come in and run new wires to the three most important spots for me. How old is the house? 1956. Was it original wiring? Do you have any way of telling?

Yeah, it's original wiring. It's real scary. It's not good. It's knobs and tubes and cotton insulators. We're... If my employment situation was slightly more stable, we would have been firmly in the let's rip all the walls out and redo everything era of this house. Yeah, but anyway. Got it. So, yeah, I had him run. 20 amp circuits to my pc to the tv area and to the garage where the servers and like network stuff all all hooks up okay but then the ups

I didn't need the UPS after that because I was using the UPS to count for janky apartment wiring. Just for that. You don't have any concerns about like in a power outage shutting things down gracefully and stuff like that. I have concerns, but I figure it'll work itself out. Probably, you know. It'll work out. There are in a number of different possible outcomes. So here's the thing.

At the time I made this decision in 2007, UPS replacement batteries were still really expensive. There wasn't a lot of lithium ion or lipo UPS stuff. It was all lead acid batteries and they were charging. exorbitant prices for them because they could yeah and to be clear it is still mostly lead-acid batteries yeah like what like what goes in your car although it seems like i think we talked about this is

Really a lot of follow-ups from last month's patron episode. I think we talked about the LifePo4. It's like lithium, iron, potassium something. I don't know what the four is. Polonium. No, it's not polonium. Probably not. It's phosphorus. Would you put a nuclear UPS in your house? I mean, look, I read a lot of Tom Swift books when I was a kid. The power of the nuclear revolution was promised and has never been fulfilled.

Would you have a personal nuclear reactor? Depends on where I live. If such technology existed. Like if I could spend a hundred grand on a thorium reactor for my house and never pay for electricity again, yes, I would do that in a heartbeat. Fuck PG&E. Fuck them to death. Yes. Fire them into the fucking sun. Yes. I think after last month's events, I think we can agree on that. No, it's not. No, it's 2020 when they burned down a bunch of cities that did it for me. Yep. Also that. So, okay.

so yeah i stopped i basically bought a roundabout ups is when i first moved here and then seven years later when the batteries were shot and they were like it's gonna be 200 bucks per to update the batteries wow and also probably the ups's i had weren't beefy enough because the computers were drawing a lot more power uh i i was like i'm not going to do this anymore i just paid once to have the wires run yeah that makes sense yeah uh all right shall we just step through my

Yeah, what's up, Brad? Okay, so I've got a cyber power. It's a 1500 VA. I've never understood quite what the VA... Volt amps. Volt amps, is that what that is? I'm just bad at electricity. I just... I've had it explained to me. I generally know that voltage is the size of the pipe. Wattage is the water running through the pipe? I always go to the water analogy, and I just can't keep it straight. Or is amperage the water running through the pipe? Amperage is the...

Voltage is the pressure. Is watts volts times amps? Is that easy? Aperage is the size of the hole. Okay. Is the size of the tube. Okay. Every time we talk about this on the podcast, I get it wrong and then a bunch of people yell at me. I think maybe my mistake is always going to an analogy as opposed to I should probably just sit down and look at equations and they'll actually understand the relationship correctly. No, I was right.

Voltage is like the water pressure. Amps is like the volume of flow rate of the water. Okay. And then watts is the combination of those two numbers. Yes. I did know that watts was kind of the final output measurement. But anyway. I want to say, is it correct? It's not correct to say that volt amps is like directly equal to Watts, right? I don't, I don't ever understand why they did volt amps for UPSs, honestly.

Maybe I'm not a suitable UPS owner. If I'm asking questions like this, I don't know. I mean, look, I'm, I'm anyway, I believe in you. It's a, it's a cyber power. It's 1500 VA. It's one of the little tower units, you know, the kind of tall. Okay. Thin. Made to sit on the floor behind the desk. That to me says it's 1500 watts. I'd have to look at the specs. I think it tops out at like 1000 watts or maybe 900 watts out total output. Okay.

Anyway, I've had it for six years. We talked on the patron episode last month's powder power outage in San Francisco revealed that those batteries are beyond done. Okay. On your $200 battery replacement thing, I got a new pair of lead acid batteries that go in this UPS for $45 shipped. Toop.

two batteries? Yeah, it takes two. It takes two batteries. This model does. I think there are some like I was looking at an Eden that takes three. So that's are these official cyber power batteries? No, the official cyber power batteries. Yes, the official cyber power replacement.

cartridge thing is like 110 bucks or something like that. That's still more reasonable for something that lasts like seven to 10 years. It's not bad, but I got these from, I pulled up a YouTube guide of a guy replacing batteries on his, and these are expert power.

which is a company based in LA. And he said he's had good luck putting their batteries in cyber powers in the past. So I gave it a shot. Anyway, they came real quick aside. Once you love it when something is okay. Okay. You know how things are always more complicated than you think they're going to be. Yeah. Don't you love it when the once in a blue moon something is actually way less complicated than you think it's going to be?

I don't think I've had one of those happen in so long that I can't remember the last time it happened. Like this guy had the same model as me, but I guess the previous revision because his process for swapping with aftermarket batteries and not using their kind of pre-made tidy little.

official cartridge replacement you have to crack open a cartridge your old cartridge to put the new batteries not like breaking plastic or anything but you do have to they do have like an like an intermediary sort of mount that you have to mount them to and get them to line up right to fit into the model and not

It's annoying. And I was like, all right, that looks like a little bit of work, but it's worth saving 60 bucks. It's 10 minutes of work, to be clear. Yeah. Anyway, the batteries get here. I open mine up and it turns out they have dramatically simplified the design.

as of whatever revision I have. And they just slot right in, in 30 seconds. It was nice. So that's just, it was, I was like bracing myself for like, ah, it's going to be a pain in the ass to have to like try to route the wires and get it to fit. you know, their little custom proprietary mounting system and get them to fit in there. And it turns out, no, they actually just went to basic slots that these batteries go in. So whatever.

That's fabulous. So I replaced the batteries. Those were fine. Put it together. It charged the batteries. It all seemed good. Three days later, I was sitting here at like 730 in the morning on the computer and gearing up for work. Everything on the UPS just went dark. Like, Oh no, I heard a click under the desk and everything that was hooked up to the UPS just died. And I got under there and had to like power cycle the UPS with the power button and it was still at a hundred percent charge.

But it had just turned itself off. Was it charge of lies or was it a safety thing? Did you get a surge? I don't think so. I have no idea. It had never done that in six years. It's weird. You got the bootleg batteries and it stopped working. Of course, the immediate conclusion was that something about the batteries was amiss. But I had tested it after I put the batteries in. It ran fine off those batteries. It showed roughly expected capacity in runtime.

And then I got to Googling and found large numbers of people with similar cyber power models complaining about the exact same problem. With the aftermarket batteries or just like after the period of time? Just in general, just, hey, mine's three years old and it started doing this. I went on the kind of networking NAS server channel on our Discord.

And asked, hey, does anybody have a cyber power? Have you ever seen anything like this? Like instantly somebody responded with the same model that I have saying like, yeah, I've got one of those. It's done that twice in the last six months. I'm replacing it soon.

Yes, we when I worked in the university. IT department a very long time ago. We replaced UPSs after five years, I think. Yeah, so I have become aware that just the unit itself needs to be replaced eventually. It's not enough to just replace the spent batteries because capacitors wear out, etc. et cetera. Search protection is not permanent, et cetera. But I was hoping to get at least one more run of batteries out of this thing. But...

So what kind of UPS are you going to buy now? Well, I was definitely super pissed that day. I was all hot and heavy looking at APCs and Eatons and all kinds of stuff. I was just like, fuck this thing. I'm getting a new UPS right now. But it hasn't done it since. Oh. So I guess I'm going to chill until if it does it again, it's gone. Like a UPS that turns itself off and kills everything connected to it is kind of like an anti-UPS.

Yeah, that defeats the whole purpose of having the gentle shutdown. Especially when it has a fully charged battery. But there was one person in a Reddit thread I found who had pulled theirs all the way apart and found some questionable wiring or something. I don't know. I don't know. Some people on our Discord, though, said, hey, this type of thing can also afflict APC or other common brands. It's not necessarily exclusive to cyber power. I don't know. Anyway, I'm in a holding pattern.

Waiting for that bizarre behavior to happen again. Well, I mean, you know, at least you got something to look forward to. Well, all of that is a preamble to what happened yesterday, which was the main event. Yeah. Which is, I tripped a circuit breaker in this room for the first time in a very long time. Sorry, I phrased that poorly. I tripped a circuit breaker for the first time ever in this room.

Okay. But ever is a very long time of living in this apartment. You've lived in this apartment for as long as I've known you. Like way too long that I've been in this building. So. What were you doing when you tripped the breaker, Brad? Yeah, I think I'm going to say the very warm straw that broke the camel's back was the space heater that I was running at 85 degrees. Yeah, that would do it. Probably. Those are usually good for...

cup 1,200 watts of draw. The little ceramic guys. Well, I don't know. This is like plastic and metal on the outside of the housing. I don't know what's... But it's like... it's like the size of a toaster yeah it's a little bigger than that yeah those usually pull at least 500 watts and up to 1500 i have one of those that's good for 1200 watts easy really okay i had no idea they pulled that much

yeah so your circuit on a 15 amp circuit it's 15 times 110 so that's like what let's see if every calculator installed here 15 times 110 1650 watts man theoretical man so your 1500 is at the max because there's there's some you know you need some slack there okay well so that that probably would have done it i mean i've got two computers i've got two computers running in here

Yeah. So I've got a couple of basic electrical wiring questions I'm going to run by you here as I think about how this went down. I don't know what the NAS polls itself. I've said recently there are 10 hard drives in there. That's a 12600K. This desktop PC is a 14900K with a 4090 in it. I was playing a game at the time, which may have actually been the thing that really pushed it over. My machine will hit 600 watts.

playing a game. Okay, this was not a demanding game for what it's worth. This was a quarantine zone, but I don't know that that matters. So, if you're playing the game, you're on a 14900? Yes. It's power limited to be clear. It's undervolted a little bit, right? It's not undervolted. I've got the PL2. Intel's got the PL1 and PL2 settings for both burst and sustain power draw. I've got that capped.

Effectively, the chip can't pull more than 185 watts. Okay. So I can tell you at idle, these two computers and everything else on this UPS are pulling about 275 watts. But that's at idle. nothing happening on this desktop and I'm me crawling under the desk and looking, I could have fired up a game. I just didn't think to do it. Running a game is definitely going to add quite a bit of draw to that.

So, yeah, your video card alone will crank up and give you another few hundred watts minimum. So, okay. So, yeah, you definitely hit 1600 watts on the if the if the space heater is plugged into the same plugs. No, no, no. So that was one of the questions I was going to ask. It's plugged into an outlet across the room from from here. But but so that's that's one of the questions I have.

okay first question are the two outlets on the same wall plate always guaranteed to be on the same circuit they're not um it depends on the way the plugs are set up like yes unless somebody did some specific screwy stuff okay something special if they if they went to the hardware store and bought the dollar 99 plugs which is what those cost usually yeah then yes okay they're the same they're the same circuit for sure i'm gonna say in this building definitely then yeah like

Unless you specifically do something to make that not happen, I don't know of a way to make that happen. How about in the same room, though? How common is it to have outlets in the same room that are on different circuits? It's wildly based on construction time and all that stuff. Your building was built in the 70s.

70s or 80s, I think. 81, I want to say, is when it was built. Yeah, so it's almost certainly one circuit for all the plugs in the room. Okay. Got it. These days, modern code is such that they... like i think in depending where you are like the code where i am is

every five feet or six feet or something there has to be a plug now because everybody's plugging a bazillion things in oh interesting when our houses were built in the 80s you think about what you plugged into a house in the 80s and it was like some floor lights in the tv yeah maybe a stereo maybe yes maybe

So I didn't know they regulated that sort of thing that specifically. Yeah. And it's more in the kitchen. So in the kitchen, you have to have them closer and you have to have if you're if you don't have specific stuff set up, you have to have ground faults on them and all sorts of stuff anyway.

fun yeah um you it's easy to find out though so take two of your floor lamps plug them into all the outlets in the room and then go flip the breaker off and see which ones turn off yeah so i frantically is maybe the wrong word but I somewhat urgently had to go around and find the circuit breaker which is probably knowledge I should have had before yesterday well I mean look You've lived there for a long time and had to know it's not. The space heater is the great tripper of circuit breakers.

Yeah, I definitely. Toaster ovens, space heaters, microwave ovens. It was extremely cold in here. Electric dryers. In my defense, very cold. Was it really? It was really warm down here yesterday. Yesterday morning was very cold. Oh, okay. But like. To be clear, tripping a surrogate is good, right? Much better than burning the house down. That's a protection mechanism, right? It's not something to be concerned about. It's good that it happens.

The reason it trips is because you're trying to make more electricity go over the wire than the wire can hold. And the wire's solution to more electricity going over it than the wire can hold is, I'm going to get real hot.

So, yeah, it's a good thing that the circuit breaker trips. Okay, so here's where we get into a bit more tech before we wrap this topic up. Nut. your server, your nut server. Network UPS tools. Yes. I've been meaning to like audit that setup because it's neglected. It's bit rotted if you like. Mm-hmm. It's a good bit rot. Some of the clients that are supposed to be connected to it and monitoring.

the ups are not because i have gotten lazy about making them stay anyway but meaning digging that thing anyway here's the here's what really got weird yesterday here's here's the thing i am now pulling my hair out about okay the the trip circuit breaker happened in the morning power to these outlets was off for like 20 minutes. It ran the battery down a good bit before I shut down the two computers down to like maybe 30% or something.

But I flipped circuit breaker in like 15 minutes. Like the UPS was back on wall power and charging again pretty quickly. And I turned all the computers back on whatever. That was probably like four or five hours after that.

for one thing, I turned on like the PyKVM and the NAS, which are, which are monitoring nut. Okay. And they would turn on and then immediately shut themselves back off. Like they would get through the boot and then they were getting the signal from the, from nut to shut themselves back down again.

As if the machine was off power. Yes, as if the power was out and the UPS was draining and was at a low battery state. So it was like... you know the pykvm has all the tiny little oled screen on it yeah i could literally sit there turn that thing on watch it boot and like show its ip address and like its little status message on that screen

And then the nut server turned it off for about three seconds. And then it would immediately shut down and go to the halted message on that little screen because it was just the clients were being instructed to turn themselves off. Then I dug into the nut server logs and I found this.

UPS status set. That is like one term with underscores in it. Seems that UPS is in OL plus discharge, D-I-S-C-H-R-G state. Okay. Is it calibrating? Perhaps you want to set online discharge underscore calibration option. Note that some UPS models, e.g. CyberPower UT series, emit OL plus discharge when, in fact, offline slash on battery. Perhaps you want to set online discharge option. That's what was in the logs.

This seems like a nut server configuration thing specific to your UPS. It 100% is. I think the OL, first of all, OL plus discharge clearly is online plus discharge. Yeah. Did you? So, well, so, so I believe that I think is a like status being emitted by the hardware in the very little. Yeah, that's right. In the, in the extremely little, I have had the energy to look into this so far.

I think OL plus discharge is a hardware code. That's something that the UPS emits. Yeah, because it's connected with a serial port or something to the Nuts server, right? In my case, it is USB to... It's just USB to USB in this case. It does have a serial port on it, but I'm not using that. But you're right. I believe that is a hardware status code that is then up to NUT to interpret and act on. Yeah. And that's where...

That's where I'm starting to get into the do I have the energy for this or should I just throw this thing in the trash? And by that, I mean, he wasted and forget about UPS is because you quickly find that like every company. It emits these codes differently. They mean different things between companies. They show up in different situations and they mean different things. Diagnosing what your specific vendor of UPS means by a thing and configuring it to handle that.

That seems annoying is going to be a lot at one point. At one point, it was one of the situations where I was looking through like GitHub issues and stuff. And I, if I'm not mistaken, somebody linked to source code. Like, oh boy, have you, have you, I'm sure you've hit one of those.

before right yeah i usually just move on and do something else to solve the problem like that that's when you know you're really in the shit where the only remaining avenue for you to diagnose what's going on is somebody links to like a literal c file on a github repo oh yeah and it's like At best, you are combing through some comments in that source if they exist. Otherwise, it's down to your ability to read C to figure this out. I don't know if I'm quite there yet, but we'll see.

When you were working on the UPS, did you perchance upgrade the firmware on it? No, but I meant to look into that. You're reminding me. I meant to look for cyber power firmware updates yesterday and forgot. Why do you ask? Well, because I wondered if you updated the firmware and they changed the way they handled the situation. No, I have not updated it. The power doesn't go out all that often. So you may not have had.

a power-off situation that caused something like this to happen. Maybe not. Like I said, it was only out for like 20 minutes. But the thing I don't understand is that the batteries were when everything went off yesterday. Also, so yesterday it went off and immediately turned itself back on. Yeah. And that's when I looked in the logs and realized like, okay, it was doing some kind of aberrant behavior here. But when you looked at the front of the nest, did it show it was charging again?

The UPS, maybe? The UPS, yeah. Yeah, it had been charging all day. I had looked at it periodically. It had been charging the batteries like it was... I don't know about that. That's weird. Like, like these states that the nut log suggested online discharge underscore calibration makes me wonder if it was in some kind of weird.

Was it in some kind of weird calibration? Do I need to calibrate these batteries? Was it in some kind of self-test mode because the power had gone out? I don't know. It makes sense that it would be calibrating given how new the batteries are to it. Maybe, yeah.

I mean, I would assume that we put a new battery and you have to run a calibration on it anyway, especially with your aftermarket batteries. But I don't know. That's weird. Is it still turning your stuff off or did this resolve itself? No, this happened once yesterday. And then I.

I rebooted the Raspberry Pi that Nut runs on and it has not given that state again in the logs. I mean, that seems like it fixed itself, man. I guess. That's kind of where I've ended up with both of these situations where it...

shut itself off last week and now the weird behavior with the power outage yesterday. It's just like... Just, you know, consider yourself blessed. Is there some kind of... Everything's coming up, Brad. Like, what's the thing with the trough of despair, the something-something hype cycle? Oh, the slope of, you mean when you leave the trough of despair? Yeah, well, I'm just saying, you know, like the idea of graphing various psychological phenomena. I was going to say that we should graph.

The slope of enlightenment is where you're at now. So we need to come up with a similar like declining slope for urge to fix a thing due to like proximity of problem. You know what I mean? Yeah. Like the day something like this happens, you're like. running around like a chicken with your head cut off, like they say back home, just going, I've got to fix this. My UPS can't be this unreliable. I've got to fix this. The next day, you're like, well, it hasn't done it again.

maybe i'll see how it goes the day after that you're like well okay now i don't care you know what i mean i i have uh you know let's just say i've been cultivating a healthy embrace of entropy you know i feel like um You know, I turned 50 last year. I've spent a fair amount of time working against the thermodynamic equilibrium. Yep, same.

And maybe it's time to, you know, let the entropy take over. I mean, to an extent, this being my work machine and doing a lot of recordings on it, it's nice to have that protection. It's probably fine. If the power were to go out and at least have time to gracefully stop the recording and make sure I don't lose it. Look, I'm going to knock on wood. We've made 321 episodes of this podcast plus about 60 more patron episodes.

Uh-huh. Almost all of them were recorded sitting at this desk. Sure. It's been fine. I do a fair number of hours of recording a week at this thing. That's true. You do do more than I do. I just... I don't want to have another huge box here that's big enough to drive this PC. Yeah, I definitely get that. Yeah.

I don't know. I think I'm, even though it's barely 24 hours since this last incident, I think I'm back in a holding pattern again, but I think like one more weird thing with this thing and I'm probably going to be in the market for a new UPS. I gotta say. I'm a little surprised at your willingness to cut this UPS loose, given the tight grip that you've held on that DBX, which has been way more faily over the last year. That is a completely different situation. Okay. Okay.

Again, as I said, I spend a lot of time recording at this desk every week. That's true. The overhead of having to get new equipment in, cycle it out, redo the wiring and everything. A lot when I just need things to work and when they work 90% of the time, it's very easy to, especially when the 10% that it doesn't work is diffused throughout the day or the week or the month. Yeah.

It seems to be mostly concentrated in when we're recording this podcast. Well, I've now found that hitting it is much more effective than power cycling it. The classic solution. I am going to RNA that thing. I will get around to it eventually, but it's also...

I need to order the new stuff and have it ready to install before I send the old thing off. Well, okay. That was just a long winded way of saying I'm in a weird state with this thing and I probably need to do a lot of nut documentation reading this weekend. I think you just brought this up because you wanted to talk about your nut server and your nut logs at length. There's also APC UPSD, which I think is a competing product. Oh.

If I'm not mistaken. I think you'd have to get an APC UPS to use that though, wouldn't you? Here it is right here on the Debian wiki. APC UPSD. Maybe I'll switch to that. Yeah, there you go. Instead of nut. I don't know. You got to get off that nut. Yeah, last thing I'll say, I know you've got one more topic to talk about here, but I availed myself of something I heard you talking about on the Dual Boot Diaries, which is...

which is journal D's facility for storing all the logs from previous boots. Oh my God. It's the best thing. Is it not? It's very good. Yeah. Very good. For folks who don't know this, you can put a, is it just dash and then the number of boots back you want to go? No, so it's journal CTL is the command. Okay, hold me back up. This is a Linux thing.

Yeah. Journal D is a component of system D. It's the thing that collects all the system logs and makes them browsable, searchable, whatever. If you do journal CTL dash B. Dash B and then a number. So if you do just raw dash B, it'll give you the current boot. It'll give you all the system logs for the current boot. But if you do dash B and then minus one, it'll work backwards. So if you use negative numbers, minus one was the previous boot.

Minus two is the one before that. So when you have a crash or something hangs your system. Yep. you can get into the previous set of logs and it just stops at the crash, which is incredible. So when you've got all these freak power events that I'm talking about, I'm able to go through successive older...

sets of system logs and see what the hell was going on. It's very good. I started to say if only Windows had something like that, I guess the event viewer is kind of that. The event viewer doesn't give you clean stops and starts. You have to keep scrolling. Event viewer is not fun to use.

Let me talk to you about my cascading comedy of errors failure on Monday morning. Monday morning I got up early and I was just hanging out. I was watching Pluribus in the living room before everybody was awake because that's a... dad only show. Really? Gina doesn't. It's too creepy for Gina. Oh, well, creepy as well. We haven't started it yet. Creepy is not a word I would have expected to be applied to that show, I guess.

Not what I was expecting from the marketing, but it's worth watching. You guys will love it. It's very much up your alley. Okay. I think I'll probably watch it with the kiddo at some point. I got to make sure it's not too adult by the time we get to the end. But anyway.

I was watching and the internet seemed to be working fine. And then my daughter woke up and was like, hey, I can't log into my class portal. I was like, well, why not? The internet's working. I'm on Apple TV right now. And she's like, well, it's not working. So I loaded up my phone and I tried to go to the internet and nothing was connecting. I was like, oh, crap.

okay, what could this be? Is the internet down? No, the internet's not down because Apple TV's working, blah, blah, blah. It's probably not DNS because I was able to connect to Apple TV, yada, yada, yada. Sorry, nothing was working at all. Some websites were working and some websites work. So when I went to Google, that worked. When I went to something else, it did not. Something I hadn't been to recently, it did not.

So some Google results, I was able to see the results in Google and then click through the page and that didn't go anywhere. Okay. That seems telling. So eventually I was like, I did the whole.

fucking dns haiku you know the uh it's it's not dns yeah there's no way it's dns yeah it was dns it was dns yeah and that led me to my pie hole and since we spoke last about the pie holes i had a spare pie three here and i set up a redundant pie hole okay But my Pi is also my DHCP server, and obviously I'm not going to redundantly set up two DHCP servers on the network because that's super no bueno.

I was doing the whole thing and I was like, oh, this is great. Now, the cascading series of failures. This is a blameless postmortem. We're not going to talk about who made the boneheaded decision here. But I set up the second. Raspberry Pi, and I did not import the DHCP settings. I just copied the list.

uh the you know the gravity lists that i use to block ads for those dns servers i keep those synced up and i think dhcp is opt-in on pihole right dhcp is often on pihole and i have a bunch of static ips set up because of course i do um I have a lot of machines that like talk to each other and it's nice to have static IPs for those. Static DHCP leases? All my static IPs except for the pie hole are assigned by the DHCP server. Okay.

So I set up the second one and I set the main one that's the DHCP server up as the primary DNS on all the machines in the house. And the secondary one is the secondary DHCP server. Now. Wait, secondary DHCP? Sorry, secondary DNS server. Sorry. Okay. And how are you just to lay the groundwork? How are you advertising or telling clients what the DNS servers are? Is that just in your...

On most of them, it comes through DHCP on a handful of its manual. Okay. The IPV6, the machines that have IPV6 turned on, I have to do some shenaniganry to make them use the... The pie hole DNS instead of the AT&T DNS that's advertised by my ISP. Got it.

And this is all it's all set up like this because I have to use the stupid Motorola AT&T router and and you can't you can turn off the DHCP server, but you can't set You can't set the DHCP server on that thing to assign your own DNS servers because they don't want you to do that. Okay, what is your router besides the AT&T one? I'm just using the AT&T one. Oh, you don't have another one. I only use the AT&T one for routing.

All it does is route. The DHCP server is turned off. The Wi-Fi is turned off. All it does is route and have a couple of ports that are open. Got it. So. Apparently, at some point in the indeterminate past, the backup piehole had shit the bed and stopped working. Okay. I didn't notice that because it was set up as a secondary DNS server on all of the clients. Okay. And the primary was working, so it just never came up.

And I didn't I wasn't clever enough to foresee that this was going to be a problem and think. oh, I need a way to track and see if this DNS server is running, right? I don't have a dashboard or anything like that. I'm not checking it regularly. I don't even look at it every... I try not to think about it except for once a month I sync up the... any changes to the gravity list, which I don't really make that often anymore. So when the primary, when the hardware

And the primary one failed. And of course, I've done all the best practices. Like I have all rammed a log and all that stuff running on them. So I'm not doing a bunch of rights to the SD cards or anything like that. Sorry, what hardware is the primary? A Pi 3.

Okay, and the Raspberry Pi 3 for the secondary? They're both Raspberry Pi 3s. Oh, interesting. Okay. I mean, it's just, I had the Raspberry Pi 3 that's the secondary was my old Home Assistant server before I got the Home Assistant yellow. Got it. So the primary conked out, and when I looked at it, my initial reaction was, oh crap, it's probably the SD card, because that's usually what it is when something like that happens on a Raspberry Pi. And I plugged the SD card in and it was fine.

And I was like, this is weird. And I unplugged. I dismounted and plugged it back in. And I was like, it's still fine. And did a few times just to see. Because sometimes, typically when they fail, they're wonky for a little bit. And it took me a shocking. I'll skip to the chase here. But the actual problem was that the primary hardware has has conked out finally really running non-stop for like a decade So it's not surprising how does my three that old? Yeah, it's pretty old

I think I set that up while we were doing the podcast. It's probably only been six years. I'll look up when it came out. What do you mean by conked out? How do you know it's dead dead? It don't boot. I haven't plugged it into a monitor. But it was getting boot errors, and sometimes the Linux kernel would boot, and sometimes it wouldn't. Interesting. Sometimes I could remote into it, sometimes I couldn't. Did you try it with multiple SD cards?

I tried it with multiple SD cards. I tried it with a clean Raspbian install. Wild. And I've never actually seen that before. This is a new one for me. Do you have any cooling on it? No, of course not. It's a Raspberry Pi. Well, I mean, the four and up definitely. Yeah, yeah. Cooling is recommended.

Yeah, if it was a four, I'd definitely have a heat sink on it or something. But I guess the three was like kind of cooling optional. Yeah. And like in fairness, it's lived in my garage, which sometimes is 100 degrees and sometimes 40 degrees. So, you know, like, you know, I don't know how hardened it is against the extreme.

I'm within the range that's allowed, but if it ever got above 120 in there and I have to turn it off, I think. So the TLDR is before I realized that I went through the whole process of reinstalling the Raspberry Pi OS. I did realize I'm on a now many years unsupported LTS version of the Raspberry Pi OS, which is a Debian base.

So I have to figure out the upgrade process for that. And it seems like the upgrade process is, yo, just do a reinstall. Yeah, so Raspberry Pi officially does not support... Full upgrades in place. Full upgrades between Debian versions, which is kind of weird to me at this point, but they absolutely... You could try it. It might work, but they officially say, no, just go wipe it and reinstall. Well, in the...

So the reason I brought this up and I thought it was interesting to talk about is I was really surprised. It's surprisingly difficult to set up a Raspberry Pi OS with a static IP address. if you're running it headless all the time it turns out you can open up the sd card you can mount the sd card with something that recognizes the file system and you can manually edit the etc slash whatever

I can't remember the name of the file off the top of my head, but it's the file. There's a file that systemd looks at. Is it resolve.conf? It's resolve.conf. Thank you. That's for DNS. So that actually, that creates systemd. Not only does that predate system D, that predates Linux. Like, that's just a Unix thing. It's just a Unix file. Resolve.conf is a broad Unix thing, but that's typically, or traditionally, that's where the DNS server addresses live. So, yeah. Because I think I'm getting...

let me get down my rabbit hole here, but I think like that's traditionally like C programs are looking at that. I think like the, that might even be built into like the C standard library or something to check there first for, for, for namesaver resolution and stuff. I'm not a hundred percent sure, but anyway, I'm. I like history of Unix stuff, but that's a pretty old one. So it's really hard to get a static IP set up. I had to put it on etcdhtpcd.conf to get that.

to run DHCP CD is like several Raspberry Pi OS's ago if I'm not mistaken you were definitely on an old one I think they I think they stopped that's one of those things sorry to turn this into the dual brew diaries but that's one of those things where there are

at least half a dozen different ways to configure a network adapter in Linux. It was maddening. And that's one of them? Yeah. The thing about resolve.conf is a lot of those ways to configure your networking in Linux will take control of it now. Oh. Yeah, of course. Like Network Manager, SystemD, NetworkD, like they will edit that file for you and in fact generally tell you not to touch it.

If you try to manually edit it, they may just overwrite your changes instantly. Which was the problem I had over and over again. I ended up having to remote into that machine using the IPv6 address. It was assigned automatically. And I was able to get to it there. And there I was able to make the changes. But it was a fairly substantial hassle. And the thing I'm going to say, the reason I'm talking about this is setting up the pie hole was relatively easy.

Fixing the pie hole when I couldn't resolve DNS addresses was an enormous pain in the butt. I've run into similar situations before. So I'm going to suggest that if you're doing this, it's worth doing the failover.

And when you do the failover, make sure you pipe your DHCP stuff over so you can flip the switch on the failover if the first one goes down. I didn't expect this to be a problem. Of course, I realized that this was all happening when my daughter had about 20 minutes until her class started. And Vinny and I were an hour from starting the next Lander stream on Monday. So I did.

Everything was working by the time the stream started. My daughter missed the first five minutes of her class. We were all good. Raise under fire. Yeah. Yeah. Like I. I've got to go through and one of my projects for this weekend is to get that stuff in order because I kind of hacked it together. And then at one o'clock in the morning, one night this weekend, I'll fix it when nobody's awake. Yeah. I'll advocate for blocky again here.

You might look into it. It's command line only. It uses a single YAML file for configuration. It's definitely not web dashboard like PyHole. It's much more stripped down. Does it do DHCP? It doesn't. It does not. No. So you actually, yes. So it may be not good for you if you don't want to use, if you don't want to turn that back on, on your router. I do not. Cause it won't let me assign stuff dynamically on the, like, yeah. Anyway, but yeah. Blocky has some nice, simple options for fallback.

If it can't read the ad block list, for example, you can tell it to just pass straight through to the upstream DNS and not. Oh, that's good. So like it won't. If something about it fails, it'll just pass everything through and not prevent you from using the Internet. That would not have helped if.

Like, say, both machines running in both instances were dead or whatever, then you're still screwed. I'm going to advocate for one more thing before we go. I, a few years ago, bought some of those TP-Link smart plugs to give you live power. uh data on on your smart plug switches uh you obviously don't want to get the dimmer ones if you're plugging a pc or something into it because it'll blow them up but you can get a four pack for like 30 bucks at amazon dude i should do that well

And it's worth buying a couple just so you can kind of, because you can hook them into Home Assistant and you get a live dashboard of whatever you have plugged in. Like I'm never going to use the smart plug part of that on a PC necessarily, but it wouldn't be nice to have the monitoring part. Yeah, and I'll tell you, it's nice to use the smart plug on the PC when you're on vacation and want to have all the draw for your entire desk turned off and stuff like that. Yeah.

Like I said, I bought a couple of them. I bought a four-pack when they were on sale a few years ago, and I just bought another four-pack the other day. The ones that I got are the TP-Link, the Tapo P110s. okay i think they have a matter version as well it's the 110m um the sorry the 115s are the wi-fi the 110s ms are the matter ones how do you how do you spell tapo is a t-a-p-o

TP-Link's weird because they also have the CASA line of smart devices, and I think they make smart plugs under both names. Oh, you know what?

I just bought the top tapos. The first ones I have are the Casas. Okay, yeah, I don't know why they have two different lines of smart plugs, but it doesn't matter because the government will ban the sale of those devices soon anyway. Yeah, so get them while you can, I guess. Yeah, I've, since I... I kind of standardized on TP-Link smart plugs since I started buying them because I found them to be quite reliable and easy to set up.

They also do power strips that each individual outlet is controlled. Man, that's pretty nice, too. They're kind of expensive. Yeah, yes. Every time I've looked at those, it was like, oh, that's a little more than the convenience would warrant.

but if you're doing the energy monitoring, it's, it's pretty handy. So, yeah. Uh, anyway, I guess that'll do it for us this, this month or this week rather. This was probably a pretty deeply technical episode. Hopefully if you liked it, let us know. Yeah, and if you didn't, also let us know, I guess. If you needed a little more context, let us know that too. Yeah. You know, when you're deep in the shit, sometimes it just pours out. Sometimes you just got to get in there and talk about the stuff.

Well, so just conceptually, I like showing both the good work and the bad work. We learned how to do this stuff because we made mistakes along the way. The DNS thing going down was... a result of me making a series of fuck ups. Yep. Yep. My UPS stuff is kind of similar in some ways. Like, yeah, I feel like this, if anything, this episode is maybe a good object lesson in the ins and outs or rather ups and downs of like.

running a lot of home infrastructure of this type well yeah and i think i think there's a real argument for sharing the learning like I made the joke about it being a blameless post-mortem because I'm the only one that cared really. It was a minor inconvenience for everybody else in the house. But it's worth thinking about what happened when this failed.

And had I gone through and at some point just unplugged the main like so so folks know the reason it was doing the reason I was getting in up and down behavior. on the primary for the failed hardware is it seems like it was it was feeding back dns queries for stuff it had cached but wasn't going out and getting new dns uh information from

the broader internet. That's why some stuff was working and some stuff wasn't. So if it was in the cache, it was fine. And if it wasn't, it just wouldn't work. And the eventual solve, I don't think I ever actually said this, but I just swapped the card into the secondary one.

And then plugged it back in and it was, it booted up fine. Oh yeah. If they're both PI threes, you could just same exact hardware. Same. Yeah. This is the benefit of both of doing both on PI threes. Right. That's sure. I hadn't thought about that. Some of that caching behavior might also be OS dependent, like some depending on the operating system. That's true. Some of them cache DNS results in different ways. That makes sense. True. For some period of time. But anyway, yeah, like I...

You know, we all make mistakes, and a lot of times people... will hit me with questions or messages on things like, hey, you'd never do something like this. And I was like, man, I have absolutely done stuff that's that stupid or much stupider in many cases. I've rendered our network non-functional through similar piehole or blocky in my case.

issues. I think I talked about this. It was last summer. It was like the morning I had a flight where I was about to be gone for two and a half weeks. I rendered the entire home network unusable.

Like an hour before I need to leave for the airport and I could not strand my girlfriend without internet for two weeks See I did that once with plumbing. I was getting ready to go on a on a trip for about a week and a half and uh i trimmed my beard the morning before i left as i do and i was like oh man i don't it's too much of a pain in the ass to get all the beard trimmings out and put it in the trash or the toilet like i usually do so i just ran him down the sink and then about

Three hours after I landed, my wife was like, hey, was the bathroom sink slow when you left? I was like, oh, man. Let me tell you where the sink snake is. Also, plastic semi-disposable sink snakes incredible an incredible like they cost a dollar fifty each it will save you such a delightful thing to just get in there and just just

jam through whatever's clogging up your sink anyway uh now that the plumbing talk is over i guess it's time to thank our patrons and remind everybody this is a patron supported podcast we wouldn't be here without you all so thanks everybody thank you We are on Patreon. You can find us at patreon.com slash techpod, where for five bucks a month, you can support the show and get access to the discord where we learn stuff constantly and meet all sorts of interesting people who have.

Weird and novel computer problems and projects that they're working on all sorts of fun stuff And you also get access to the patron exclusive episodes where we where you can see the kind of origin story of these kinds of episodes. So. Again, it's patreon.com slash techpod. We appreciate everybody's support a ton. It helps us keep the show going, which we love. We do. And we especially appreciate our executive producer to your patrons, including Jason Lee.

nextlander.net, it's .com, Andrew Slosky, Jordan Lippet, the bunny word spiral. It's a Nine Inch Nails reference. Yeah, sure, I believe that. David Allen, James Kamek, and Pantheon, makers of the HS3 high-speed 3D printer. Thank you also so much. Thank you. We will be back next week with another edition of the TechPod. I think, is it questions next week or we have one more?

I believe next, let me triple check. Yep. Nope. Next week is in fact the last Sunday of the month. Also we skipped questions last month. Oh, we have a double dose of questions. So we should have plenty to choose from. Yeah. So, but don't let that stop you from asking more. I was going to say, you can send your emails to techpod at content.town or...

You can send your questions in the Discord, if you're in the Discord, to the Questions Seeking Answers channel, and we will answer a select number of them on next week's episode. I'm going to say, we know that we forgot the Neo Geo Pocket Mini.

And the regular Neo Geo. And the regular Neo Geo. Is there anything else we forgot to rank in there? There was something else somebody brought up. I can't remember what it was. I think somebody mentioned the Sega Master System, but I don't remember that having... We'll have to investigate that. Let's see. We didn't do any Linux startup sounds, Adam Patrick Murray. No. Nobody cares.

I mean, I care. I'd like to hear them, but I think that's out of scope for that episode. They're all user configurable to the point that I don't think there's any canonical true ones. Also, there are so many desktop environments. Like, where do you even start? Yeah, I'm not... I can't even...

I can't, they're not on YouTube. If they're not on YouTube, they don't count. But yeah, so if people have questions, if they have feedback, if they have things that they want to add or things that they think we got wrong, please, techpod at content.town or the Question Seeking Answers channel. We will be back next week with answers to all of your important questions. Every single one. But until then, please consider the environment before printing this podcast.