2.5 Admins Episode 220. I'm Joe. I'm Jim. And I'm Alan. And he were again. And before we get started, you've got to plug out five reasons why your ZFS storage benchmarks are wrong. Yeah, so we walk through the common mistakes people make when trying to benchmark any file system, but especially ZFS, and why the results you're seeing might not be measuring what you think they're measuring and how you can do it right. Right, well, Lincoln, the show notes as usual.
Something occurred to me recently, and that is that ransomware should not be a thing in 2024. In a world where ZFS and snapshots exist, ransomware should not be possible. That's how I felt in 2014. Right. Now, clearly not everyone is using ZFS because ransomware is very much a thing. So I thought it might be interesting to go through why ransomware shouldn't be a thing if everybody just used ZFS. And it doesn't even have to be specifically ZFS. You just need things with snapshots
that are copy and right. And basically, it means if you have your juicy file server fill of all your important business files, and you're taking snapshots on a regular basis and retaining those for a reasonable amount of time, when the ransomware shows up and starts encrypting your files, it's going to write the encrypted version of the file to a new place on the NAS, not over top of the original version. So it means via the snapshot, you can go and pull out the unencrypted version.
And especially like the ransomware that I had experienced with, it was a bit smarter. It didn't encrypt the whole file on every file because that would take days to go through the whole NAS. It only encrypted the first megabyte, the last megabyte, and some megabyte about 33%
through the file just enough to mostly stop the files from working in most programs. If you take out the first and last megabyte, most videos aren't going to be detected as videos and they won't play, office documents won't open, everything will be screwed, but you only had to write three megabytes for file instead of some of these files were multiple gigabytes, and it would have taken a lot longer to encrypt. Can we just go back a step? Because the obvious
answer to avoiding ransomware or mitigating ransomware is backups, right? Yes, thank you. I was waiting patiently for the chance to get to that. It doesn't really have anything to do with the way that the CFS does it. You don't specifically need snapshots. What you need is backup that you can rely on in an RTO and an RPO that are both short enough that you would prefer to go to backup than to pay some asshole money who happens to be the same asshole that just messed up all your
stuff to begin with. Right, but if you've got terabytes and terabytes and terabytes of data that is backed up, it could take days or even weeks to restore in a traditional backup setup, but with ZFS, if you've got the snapshot, you can do it in seconds. But again, we're just talking about an RTO recovery time objective, an RPO recovery point objective that are low enough
that you would prefer to go to backup rather than talking to some jerk. Now, ZFS goes way above and beyond on this because your RTO is usually going to be in the milliseconds, not the seconds. And your RPO, assuming you did something smart and you set up automated snapshotting, well, it's going to be whatever you set that up to. It might be it's hourly, typically for me. I know a lot of folks who do like every 15 minutes or every five minutes even if that works for you.
That's fine. But again, the whole point is backups. You were very right about that and a very low RTO and RPO. And if all you want to know is like, you know, how do I make ransomware not be a problem? You don't really need to know more than that. However, you satisfy those requirements and get your RTO and RPO low enough, that works. That did the trick. You no longer want to pay some jerk to decrypt all
your data because you would rather just get the decrypted copies yourself off of backup. However, I do think that we're doing everybody a really big disservice if we pretend that the only facet of modern ransomware is the encryption. In fact, especially when you're talking about larger businesses with competent IT staff that are actually supported in their mission objectives, these days, that's not really the scary facet of ransomware. The scary facet is exfiltration.
And ZFS can't do anything for you about that. ZFS can absolutely protect you from some jerk encrypted all my data. And now I can't read all my data. ZFS can't do anything for you when that jerk instead spent the last three or four months siphoning out every bit of confidential data from your network that they could possibly manage and is now threatening to publish it everywhere. There's no easy answer on that. That's why you still need an InfoSect team. ZFS is great,
but it will not replace an InfoSect department. Yeah, but the one that really brought it home for me was, you know, I was visiting my college and they were having a big IT emergency because some ransomware hit the network drive for the medical students. And it was going to take them about two and a half days to restore all the files because their RTO was that bad. And they couldn't make their RPO. They're like, how long ago since their last backup any better because the backup took
so many days to run. Whereas, as ZFS snapshot, like Jim said, you could have it happen every five minutes because it doesn't take any time to create a snapshot. It doesn't have this latency between how often you can take them. Whereas, if you're doing a traditional backup to tape, you can't do it more frequently than how long it takes to do a whole backup. And one crucial aspect of ZFS snapshots is that no one and nothing can change that snapshot once it's been taken, right?
It can be destroyed, effectively deleted, but not even ZFS itself can change the contents of that snapshot. That's correct. It would be very difficult even to build a tool to do that, to a snapshot in the middle of a chain because you've got, I said chain and I should have said
tree because that's the thing. You can follow the tree all the way back. So it's very difficult to edit something in the middle in much the same way that if you've already printed your novel and the page numbers are on every page, you can't really just stick a new paragraph in the middle somewhere. You'll have to reprint the second half of the book if you do that. You have a very
similar situation going on with ZFS or even God help us butter. Yeah. So if ransomware does try and encrypt everything on the system that it sees, everything on the file system that it sees, it cannot touch those snapshots. If it has administrative access, it could try to delete the snapshots if it was ZFS aware or malware. Yeah. Although to help with that, you can put a hold on a snapshot, which basically locks it with a, it's not encryption or a strong password. It's
more like a lockout tag you use when doing maintenance on physical equipment. So somebody doesn't turn the equipment on while you're inside of it. It prevents, for example, your automatic snapshot script from cleaning up these old snapshots if you still need them for something. You can basically put a hold on it saying, Jim needs this for something. Don't automatically delete it and it can't
be deleted until somebody does a release on that. Yeah. But if my Windows laptop is connected to the network share that is backed by ZFS, my Windows laptop has no clue that it's dealing with ZFS. It's just a network share and the ransomware will go and attempt to encrypt everything that it sees. But it cannot even conceive of the snapshots. So somebody like, depending how it's set up, the ZFS snapshots can be exposed to Windows as volume shadow copy, like previous versions and so on.
However, there's still read only their immutable snapshots are immutable period at the ZFS level. Again, in theory, if you're really, really good at math and a whole bunch of other things, you could make some malicious tool that tried to generate hash collisions to edit blocks in a snapshot in the middle of a tree. But why? There's not even any point in trying to do that. Like at that point, you would be better off just destroying everything because that you can do
if you've got root access. You can destroy the snapshots just fine, but you can't monkey with them. But you need root access on the server where ZFS is living. Even if you've got admin on the Windows laptop that's talking to the network share, that's still not going to be able to destroy
the snapshots. Correct. That will do you no good because the only way that you would have even the read only access to the snapshots from the Windows box that's just doing like an SMB share across the network is if it has been set up to expose the snapshots as volume shadow copies, then you can see the contents of the volume shadow copies in the Windows Explorer File History tab. But there read only. There's no right access being given. The File History client in Windows
doesn't understand the concept of writing to that. You can just copy stuff out of it or not, or you can destroy it, again, much like snapshot, but there's no provision to go back in time. Yeah. You can't go and mess with stuff with the client. It doesn't work that way. Even if you could, Samba on the other end is it doesn't have that concept. It's not set up to say, oh, you want to edit a snapshot. Let me help you with that. Or you want to destroy a snapshot.
Let me help you with that. That's not Samba's job. It doesn't do that. It doesn't know how to do that. Effectively, yeah, even if you just want to destroy the snapshot, you need to have root or delicate, destroy privileges for the data set in question on that box. You need to get a shell on that box in order to do it. You should not have given the person who's got the Windows laptop,
any such access. You absolutely should not. It's not always in the case of a separate physical machines like a file server in a closet somewhere versus your laptop sitting on the dining room table. In my case, usually the separation is more at the virtual machine level. The hypervisors, the big barrier for me. On one of my servers, I usually have five or six virtual machines that all think they're quite large, real boys. Thank you very much. They've got their own hardware
and their own storage and their own everything as far as they know. They're wrong about that. But the point is, the snapshots exist at a level that they have absolutely no access to or even awareness of. Yeah, but speaking of awareness, one of the interesting things about ZSN snapshots is it can help you detect ransomware earlier than other file systems in that when they start encrypting files, they're going to start taking up more space because you're keeping the old
version and the encrypted version. And so if you're not expecting to see terabytes of data getting written to your NAS, you're going to notice when terabytes of data is start showing up. Yeah, writing terabytes of data to your NAS when you're not using a copy on right file system doesn't necessarily mean you need terabytes more space on your NAS because you're munking with things in place. Whereas because ZFS is copy on right and this is going to be true for any other copy on
right file system as well. Now, to be fair, again, you do have to have some snapshots or else what happens is you create the new block and then you unlink the old block so you can still kind of chew through the space without noticing. But if you've got a snapshot, every block you change is another block you have to write. So you're not just seeing terabytes of data being written, you're seeing the volume of data on your NAS growing by terabytes in a day or two.
Yeah, and if you've got proper monitoring setup, which you should have, then you're going to notice that really quickly. And both, you'll notice the increase in rights that wasn't there before and the decrease in free space that you know is unexpected. And now I know that you should definitely have backups. Obviously. Yeah, so just snapshots on one NAS is not enough, right? You want to be
replicating those snapshots off to somewhere else. And then probably ideally from that backup, taking a non ZFS backup using the snapshots to have a stable version of the source and having, you know, that third copy be something that's not ZFS just in case. Right now that's what you should do, right? But if all you had was snapshots, which are not a backup, but if you had snapshots,
that could or probably would still mitigate ransomware, right? Yes. If you've got the snapshot on that box, even if it's not backed up, because the ransomware can't touch the snapshot, so you just roll it back in less than a second. Jobs are good in. I'm not aware of any ZFS or where ransomware right now. But if you've got ransomware on your box, it almost certainly has root. And if it has root, it can destroy snapshots just fine or destroy entire data sets or do whatever it want.
It's got root on your box. Right. But again, I'm thinking more of this kind of samber situation with a Windows laptop versus actually getting ransomware on the server itself, the file server. Okay. Fair. Well, just you said one box, only the one box with its snapshots. I wanted to be clear. Yeah. You do need to have another layer of separation. Otherwise, it's just
security through obscurity. And you're just betting that the ransomware or the script kitty or whatever won't know about the ZFS and won't be bright enough to go destroy all your snapshots, which isn't the worst bet in the world, you know, in terms of winning the bet. It's just a really dumb bet to make regardless because you don't need to take that bet. Just do better. Okay. This episode is sponsored by people who support us with PayPal and Patreon. Go to 2.5abmins.com slash support
for details of how you can support us too. Patreon supporters have the option to listen to episodes without ads like this. And it's not just this show. There's late night Linux for news, discoveries, audience input and missanthropy. Linux matters for upbeat family friendly adventures. Linux after dark for silly challenges and philosophical debates. Linux DevTime about developing with and for Linux. Hybrid cloud show for everything public and private cloud. And ask the hosts
for off topic questions from you. You can even get some episodes a bit early. We've got a lot going on and it's only possible because of the people who support us. So if you like what we do and could afford it, it would be great if you could support us too at 2.5abmins.com slash support. Let's do some free consulting then. But first just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. And if you want to send any questions for
Gemini Island or your feedback, you can email show at 2.5abmins.com. We often get quite a lot of questions about how to get started in a Sysadmin career. And we're going to cover three of them now. Riley is considering a career change from another tech related field. Jim, not that one, is a 25 year Linux hobbyist who'd like to start getting paid for it. And LC is fresh out of high school. So let's cover some of the challenges that they're going to be facing and the best strategies to
overcome them. You know, the industry has gotten a bit weird about the classic actual system in role, you know, with the advent of DevOps and a bunch of these other related things and even just deciding to call them SREs and so on. A lot of times the classic system in that actually knows how the surface work isn't a role a lot of companies hire for anymore. And I think that's going to be more and more of a problem for those companies when everybody that works there only understands
the higher level of abstractions and doesn't actually know how anything works underneath. But I think this kind of beside the point to what these people are asking about, just how to get started with a career. I don't know if I'm the best person to give advice because I kind of fell backwards through my career and started my first business being a Sysadmin and hosting IRC servers when I was 16 out of my parents' basement with a co-located server at the cable company.
And kind of through that went on to never have had a normal job and always kind of worked for myself. But I think Sysadmin's stuff is really a lot of getting to know what you're doing and through that being able to solve problems that people actually have. And so for HomeLab stuff a lot of that means getting experience with the types of problems that businesses actually have when they're
trying to deploy stuff. So if you're using Proxmyser or whatever that can make sense but you have to look at how they're going to use that in a business to do critical things versus using it for your home media or just running a couple of VMs that are toying with stuff. How do you make different components work together and how do you set up a single sign on system between a bunch of these different services or web features or whatever that is going to be more applicable to what a business
is doing. Yeah, how do you set up a whole stack and document that stack and make sure that stack is backed up and understand how to recover when things go wrong. Like these are all the things that you want to be able to do. And ideally you not only want to be practicing all this stuff,
you want to be doing it with the tools that you would like to work with. Now in some cases that's going to mean well I know that I want to work with this company and I have some friends who I can ask with that company uses and so I can practice on some of the same tools and applications that company actually uses and I can get very focused direct applications to try to get me into that
specific job. In other cases it may be a case of like well you know I have discovered these tools that I've built a stack I'm really proud of and I think that you know that could be applicable out in the business world and I'd like to bring this knowledge and expertise out there and that is also very valid. Now that's going to tend to lead you more in the direction of a career like mind mercenary sis admin because like you know what you want to do and you're just like hey
give me some money and I'll do this stuff for you too. And if you're good at it folks will eventually do that. However that career path isn't for everybody because in addition to being a sis admin you need to have at least some ability for like general face to face humaning you got to do a little bit of sales you got to do a little bit of marketing you need to be able to do some of
the bookkeeping on the back end you got to you know actually put an entire business together and to some folks that's going to sound awesome it certainly sounded awesome to me to other folks that's going to sound overwhelming and intimidating and that is not what I want at all so if that's
not what you want then if you're not choosing a specific set of tools to learn in your home lab because you know that they're applicable in a specific job or at a specific company that you'd like to apply for then you may wind up wanting to just kind of like work through everything that
you know is is out there in the industry maybe you already got an entire stack set up on proxmox and you think that's great and like you don't even want to change it but you may also want to set up the same stack on normally I would say ESXi if that's kind of a dead horse at this point
just in general you may want to experiment with other technologies to do the same thing because the the more that you can just say oh yeah I know that technology I have experienced that technology oh yeah look at you know my notes over here on you know in just on GitHub or you know wherever I
I've put my documentation but again you want to have things you can point to you don't want to just be sitting there telling somebody oh yeah I've worked with that it's the first time they're hearing of it you want to be able to say oh yeah here's here's all my notes on that like here's all the things that I've done with that and here you know lessons that I've learned and now in most cases an interviewer is not going to dive that deep into that stuff but just the fact that you can
confidently like oh yeah there it is and here's all my documentation on it yeah they're going to remember you yeah well like when I'm looking at resumes is there's a link to a GitHub I'm looking at what they did how many contributions are they regular is the documentation any good as opposed to some
people included link to their GitHub and it's like oh I haven't done anything in two years it's like well that's a great resume kept there I do want to point out Alan is a small business and small businesses hire very differently from large businesses when Alan is looking to hire he's looking
at raw resumes trying to find people that he wants to hire he's doing the kind of all in one thing if you want to hire a bigger more traditional you know corporation you don't get to talk to the interviewers yet you've got to make your way through you know the the folks in HR that are
putting out the the job ads they're you know getting a massive stuff in from recruiters or indeed dot com or whatever and they're just they're looking for you know radio buttons to be marked they're looking for check boxes to be ticked and so you got to get through those folks and now those folks
may not care about your GitHub you need to make sure that you've got whatever certificates or you know as much as you can whatever kind of experience those folks will look for to get through that firewall to the people you really want to talk to the other way around that is to talk to people
in the industry make friends make contacts if you want to get hired for longer than I've been alive and folks I'm 52 years old for longer not been alive the best way to get hired anywhere is to know people there because they can get you through those firewalls that was talking about they can
lean internally on people if they want to get you hired they can not only move you past you know some of that HR firewall to get the interview they can lean on the interview and be like look this guy's awesome like we need him we want him or her or they are you know whatever the point is
cultivate contacts go to conferences meet people give talks as a matter of fact that will get you into like speaker sponsored dinners you'll be sitting at tables and like rubbing shoulders with people and you make friends you impress people there they will want to bring you in places but to
Jim's point like even with our smaller company when we do put a post out on LinkedIn or indeed we get hundreds of resumes half of them you know applying for the you know ZFS kernel developer position and their only experience is JavaScript full stack it's like you why are you applied to this
job and so a lot of it is you need your resume especially you know it's all you only have like a page or two to get my attention and you kind of got to almost SEO it right you it has to have the keywords I'm looking for and it has to match the things we know and to Jim's point those people
in HR they don't know what any of those technology words mean they're just binary filtering of does it say Kubernetes on it or not and if it doesn't say Kubernetes it goes in the circular file and so you really got to look at each particular job you're applying to and sometimes it's just tweaking
three or four words on the resume by getting a version that's going to get past that filter and that would get you a lot further than a lot of other stuff to Jim's other point about yes six I going away there are a lot of businesses using VMware that would love to have some people
that have experienced migrating from VMware to proxmox this is something you can teach yourself in your home lab get to work right a guide post it point to it it's also a great way to get contract work if you're in for that and a lot of the time contract work will lead to much longer employment as well I can't tell you how many times I've gotten called in you know for a tiger team like one off contract and the folks there decided like oh this is somebody who actually knows what they're doing we
would like to have them back for other things and you know then then you can get approached for that and you can have longer running engagements or repeat contracts or even you know convert to salary I've done all three of those things in my career yeah I was the same thing you know
helping somebody out of buying one time and it's like oh you actually understand how our things work and you just got here we need you all the time so we've been a great customer for the last five years now and to Alan's point about the resumes and you know you've got to have the right
keyword in there if you're applying it a larger business because the folks they're looking at the first round of resumes they don't know anything about any of the stuff they're just looking is the word they're not that leads into the the question of like what do you not put on your resume for
example if you really really want to get hired in for a junior Kubernetes job and you've been doing stuff with Kubernetes in your home lab for five years you may want to really consider just putting as a bullet point five years Kubernetes experience on your resume you don't have to say
home lab you can talk about that in the interview that's not shady that's not sketchy if you get the interview and they ask you so tell me about your Kubernetes experience there's absolutely nothing wrong with saying well you know it's it's in the home lab but let me walk you through some of the
things that I've done it's fine and doing it that way maybe a better approach than actually just putting you know I've done this in a home lab right there on the resume where the people who aren't really the decision makers might see that just go oh well that doesn't count check it in the bin
at the same time don't just keyword stuff whenever a resume is I've seen recently talking about somebody's all I have experience with windows list all the version including XP it's like well that's not relevant to this job and also nobody cares about windows XP anymore unless it's something
very specialized where you would maybe want to point that out but yeah don't put a bunch of irrelevant stuff in there I don't care that you know how to use Excel if I'm trying to hire you to be a seed developer it's worse when you are trying to hire someone that to a job that requires Excel
and they claim they know it and they really don't but try to avoid listing too much completely irrelevant stuff and focus on the stuff that is going to matter if you just have a giant laundry list of skills they're going to be like well this person isn't good at anything they just have a
giant list of things and we're probably not going to believe that they say they know these things that just brings us right back to Taylor Gresemey again the length of the resume that you submit for a position is incredibly important and shorter is better like a page and a half is the
generally accepted maximum but if you can fit it on to a page and hit all the bullet points that is absolutely what you want to do believe me on that one don't feel like you just need to keep cramming extra weird crap in to make it longer and also again this should be a tailored
resume every time you give somebody a resume it should be a resume tailored for that position that you want you don't write one generic resume and give it to everybody the same way you don't like learn one pickup line and walk around the same bar all night saying it to every single person
of the appropriate gender you see in either case you're not going to have the results that you're looking for and you know you might get slapped no we've mentioned this a few times it's kind of been a through line but I think it might be worth explicitly pointing out that to learn new things
you have to do those things it's no good to just read about them you actually have to do it in a home lab on a vps whatever it is I would say go even further is you have to write about it it doesn't matter if anybody ever reads it but as you're doing it write down what you're doing the steps and
why it'll be very useful for yourself but also when you get to the end of the deleted and do it again following your instructions and find out all the places you left things out of your instructions because there will be a bunch and if you do it while it's fresh you would get those
instructions to be good and you're really going to appreciate having those instructions when you have to rebuild something at three o'clock in the morning because your brain will not be working all the way and it's not going to remember those subtle details and your documentation that does
cover them will be good but I can't express the number of times that people documenting what they were doing has helped them I have a friend who does freebies these system and work and occasionally he runs into a problem and when he does because he's documented what he was doing he can show me
all right this is what the system looked like before I started that's a step people always forget before you start making changes to a system document what it is right now I just get the dump of like z-pool status and if config or whatever IP adder and know what the system looked like
when it was working and then write down everything you did everything you did and then when you run into trouble somebody will be able to see ah there's the typo right there or whatever and to be able to help you out of it but it also means you yourself can look over it with fresh eyes
even if it's just the next day and be like ah I see what I did wrong now and get yourself unstuck even if you never read it again the act of having to create the documentation will fix the details of what you did in your brain a lot better it's the same reason you're supposed to write down
on your own notebook all the stuff the teacher wrote on the chalkboard or the smart board now or you know what have you in school it's not because you know teachers are too dumb to know about Xerox machines or about just emailing you know notes to you from their own document it's because
the muscle memory helps cement that knowledge in your head and and lets you recall it later a lot better and I will say also like if you never go back and reread your own documentation like you should start because I don't care how great your memory is I don't care how much of a natural
you are if nothing else you don't know how good a job you're doing with the documentation unless you go back and read it and part of the reason that we say this specifically about job seeking is even if you never ever need that documentation again yourself because you're some kind of weird
mutant you didn't need to take it you don't need to read it again you're just amazing well the fact that you documented it is one of the things that people are going to look for it lets them not only for one thing believe you a lot more when you say oh hey I did this thing because they can see all the steps that you did it also tells them hey this is a person who documents the things they do which is going to make them much more valuable as a team player so that's going to make you stand out
yeah I've run across a couple of people in my career who felt that their job security came from the fact that only they knew how this system worked and they wouldn't tell anybody else and it's like yeah and then they were on vacation or sick and nothing could get fixed or we had to interrupt
their vacation in order to get it fixed you're going to start as a junior system in probably but if you want to be the senior one someday you'll have created that documentation so that it's not you that has to do the steps at three in the morning it's somebody else's problem they could just
follow the steps you wrote down for them I will also warn you folks my entire career I've been hearing all those jokes you know about oh well you know they can't fire me because I'm the only one who knows blah blah blah and sometimes it's jokes and sometimes people mean it seriously but
that's people talking as far as what I've observed happening I have long since lost track of the number of cis admins who don't document what they do and you know who nobody can figure out what it is they're doing and they wind up getting forced to train their own replacement the company's
not necessarily going to wait for you to go on vacation for something broken while you were gone like they know what you're doing and they don't like it and the odds are real good that eventually you are literally going to watch your own replacement get hired and you're going to be given a few weeks
to train them and either you do or you don't but either way you're gone because that company's going to be tired of your crap don't be that guy B.O.F.H. jokes are jokes they're not a role model to aspire to right well we better get out of here then remember show at 2.5 admins.com if you want to send any questions or feedback you can find me at jorest.com slash master.don you can find me at mercenarycissadmin.com and I'm at IowinJude we'll see you next week