Hey everybody, and welcome back to another episode of Adventures in DevOps. This week, on our panel, we have will Button. What's going on everybody? So I have Jeffrey Grumman. Hey there, I'm Charles Maxwood from dev chat Do TV. And we thought, well, we had a guess not show up, and then Jeffrey's telling us about all the horrible but fascinating things that are going on on the Internet these days, and so we're going to
talk about more breaches and concerns and stuff. Is going to freak me out, but I'm kind of curious in a morbid way to hear about so I guess, I guess let's just dive in. So, Jeffrey, what's broken in the world? Oh? Man, I gotta tell you. It's been a crazy week. I think that started out last weekend and a piece of software called Case, which is used by different managed service providers, was found
to be carrying malware in terms of ransomware. And so basically any managed service provider who uses this case to like manage their customers or the client environments, all of a sudden were just locked up. And that means that they can't you know, that means they themselves are locked up, and that means that
they can't service their clients. So basically you're like, you know, if you're using an MSP that gets that it uses cassea, you're basically like you have no service, Like you have no IT service provider anymore because they're just their host. So that's how we started the you know, the week coming out of the long weekend, the holiday weekend. And then Microsoft, I think a couple of researchers found a really crazy remote code remote code execution RCEE
vulnerability in Microsoft Windows Principuler. So there's a lot there. Maybe it's just worth us a couple of minutes in talking about what that means. So if you're not familiar with Microsoft Windows, going all the way back to the beginning of time, one of the biggest things that people used to do was print, and if you weren't printing off of your own machine, you were printing off of a print server, which is very common if you're not familiar.
Printers are these big, oblong plastic devices that spit paper out from time to time or toning. Right. It's funny because these are like this is like legacy stuff, but it still is, right, there inside of every Microsoft Windows whether it's like the desktop version or the server version, they're all running the print spooler. It's a service that runs by default and it just has all this legacy software in it because it's just been around for so long.
And so some researchers have been spending time looking at that and they found this again, the remote code execution. What does that mean? That means that I don't have to be connected to the machine to be able to like literally like logged into the Windows guy to be able to exploit that vulnerability, And the exploit of that vulnerability gives me what's called system privileges. A system is basically like a service account version of like your local administrator. So it's an
elevated privilege that certainly could be used for all kinds of nefarious purposes. Interesting thing about it, though, is that that I think, if I remember right, or if I understood this correctly, you have to have a legitimate user account on the box to be able to take advantage of this, Which makes sense because the Prince Spooler is typically not something it's not like a web server that any anonymous connection can use utilize do something make requests of et cetera.
The Prince booler is only available to users of that system, so you have to have a Windows user account already. So that was something that you know, when you think about like the risk side of this. Okay, how do I prioritize or what do I think about this vulnerability? Okay, Well, if it means that somebody has to have local access or you know, an account of the box already, then maybe some of my systems I'm
less concerned about. But maybe if I have a Windows system that is exposed to the Internet, I might be more more concerned about that, especially if it's let's say, like a SharePoint server right that's exposed to the Internet, where somebody out there might have a user account on my box for whatever reason. So then I'd probably be more concerned about it. But yeah, through this and it went on for days off, I'm just trying to figure out
how we fix it. Microsoft was trying to figure out workarounds. Researchers were trying to find out workarounds. Can you just shut down the print spooler service? But that's it wasn't enough. That actually doesn't stop the vulnerability from being
exploited. I think finally, today I saw an update that Microsoft provided an out of band patch, out of band meaning not part of their like patch Tuesday, which is I think a monthly cycle, so dead it's out of band to that they just released a patch for I think for basically every version of Windows. So if you haven't gotten that, if you're not aware of this, you can certainly download it from Microsoft. But yeah, that's basically it. Then nuts all though, I think there's a lot there to talk
about. So it sounds like if you just delete all your user accounts though, then it can't use to exploit anything, right, that is correct. You know we used to talk about you know, this is way back in the bill before the bill gates famous letter about writing secure software. We used to joke that the best way to secure a Windows server was to fill the footprint with cement and then bury it in the ground. So deleting every user account on the device is similar to that approach. It is it is no,
yeah, no, no, yes, we're paid on results. Yeah. Yeah. Apparently it's been according to Microsoft and SIS, which is there's a government agency called the ci A and they sort of track these vulnerabilities across all vendors and provide bulletins and that sort of thing, and they claim that
these have been exploited or there have been exploits seen in the wild. I haven't heard of any specific breaches based on this, but clearly like something this big and this that's like sort of easy to exploit vulnerability is going to be you know, there's going to be folks out there that are basically going to be testing, you know, every system they can to see if they can
exploit it. So so yeah, because because some of the people who are vulnerable to this are used are relying on the MSPs to provide their IT support if you can't do that because they've been locked out my ransomware right right, It is the perfect storm. So again we're back to deleting all the user's
accounts is really our only option. Absolutely absolutely, you know, going back to pen and paper forms in triplicate, if you still have them in a box in your closet, you might want to bring them out stock in carbon paper this afternoon. I think that's where we're going. Yeah, I didn't know you mentioned before the show, Will that you you went you were paying attention to some of the I guess recount stuff in Arizona, and they somebody
mentioned that they should go back to paper ballots. Yeah, And I've been to a number of political I'm fairly involved in Utah politics, and yeah, there are a few more than a few people we've gone to electronic voting, especially during COVID right where we couldn't actually actually get together and vote with paper
or anything that looks like paper. And there are so many people that have brought up I just want to go back to paper because I know you can count it, and not to get political and not to go into any of the implications of how that can go wrong, right, But it's it's interesting that yeah, Chads my friend ha, Yeah, well yeah, nobody on the outside can hack the paper, but that there are humans counting it.
So, I mean, it's just it's really interesting that a lot of this really comes down to what users do and how they approach it, and at some level there's trust, right, And I think that's the point is delete the user's account, is we're kicking all the people off that we don't trust. Yeah, yeah, I mean it's interesting. Well, and I think just to round out that the discussion about voting, I mean, I once in a while, I will hear somebody talk about, hey, we should
make it even easier to vote. Let's why why aren't we doing it? Why aren't we letting people vote over the internet or over their mobile phone? And if you hadn't thought about it in the past, like these two examples by themselves, like of the vulnerabilities and just how easy it is to manipulate or to break in, should give anybody pause about thinking about having something like voting, something as critical to society as a whole, to put it online
or to put it on a mobile device or which is also online. But you know what I'm saying, Like, it's just it's so difficult to try and protect these things, especially when you talk about third party software. Right, you're running in a window, or you know, you're running your business on a piece of third party software, and you don't know, you don't
know how well that company. I mean, listen, when we're talking about Microsoft, they have and they now have a long track record of, you know, really trying to be on top of the security, and I think they've but how many lines of code are in you know, Microsoft Windows. I mean, we're talking about it enormous, right, oh yeah, chunk
of work there. So it's and and we don't know, like it's that that really is the challenge, Like, you know, you try to think about the risk to your business and how to quantify risk, and when you're talking about putting a piece of software and running your business on that piece of software, like you have no idea what that risk really is. Well, and you talk about you brought up the voting and just the security of all the things involved, but even down to the user devices, I mean,
how many of those are compromised that people just don't know about. Yeah, it's it's so fascinating to me just to see all the different levels that this goes to and yeah, what what it effectively boils down to, right, I Mean it's always talked about security versus you know, sort of the usability the convenience of it, and security always will get in the way of usability
and convenience. I mean we see that in a walks of life, you know, whether it's physical security where you have to go through a security checkpoint or something like that and you've got to take off your shoes or you know, whatever it is that they're making you do. There's always those inconveniences,
but you know, depending on what you're doing. I think that's the important part for anybody running a business, is what you know sort of what due diligence are you doing or what how are you thinking about the convenience versus the usability or versus the resilience, right, convenience of just buying some piece of software or downloading a piece of software versus the disruption the potential disruption to here the entire business, and that has to be well thought out, I think
today, especially when we're doing so much more online. I mean, remember the days when it was not that unlikely, or there wasn't that uncommon that you go into a store and the credit card swipe device thing was offline because the phone line was down or remember that, And so they would take out from under the counter, they'd take out the dow hiki that goes back and forth over the carbon right, And everybody still have those today. That happens
in your store. I was in a home depot the other day and they're like the manager was like running around the aisle saying sorry, sorry that we can't can't check anybody out right now. I don't know why there's no work
around today. People don't really have that sort of thing. So, yeah, I was at an if A country store and which is just kind of a local farm store here, and what they were doing is they were they were basically grabbing They jot down your credit card information and tell you that they were going to charge your card later, which didn't necessarily make me feel good.
But then they gave you an invoice, a copy of the invoice right right, which was also fascinating in its own way because it was like, I don't really you know, and yeah, then yeah, the charge came through later, and they're like, yeah, don't leave that invoice anywhere because if you do, it's going to be a problem, right, And yeah, what do you do? Yeah? And again I I just don't.
I don't. I think that's that's just sort of a short, shortballower or just something that has not been really thought through in many businesses, many you know, many companies, of just what happens if you know something some key piece of software that we rely upon, and sometimes it's not even something you you even think about or realize. Like let's go back to the solar winds example from right from January or or last December where it wasn't that wasn't a
ransomware that was linked back to somewhere some group within Russia. And you know, it seem like they were going after very specific companies and government agencies, but still very disruptive to your business. And you didn't even think about Solar Winds. Most people didn't even think about didn't even have an idea like who
is solar wind Who haven't even heard of this company? But it's a piece of software that so many IT shops are using to manage their own infrastructure, servers, desktops, network gear, all that stuff, which made a prime candidate for being for hacking into it and putting basically embedding your malware into that piece of software, because it's just it's ubiquitous, like so many companies use
that. So if you're sitting in the business office, a business side of things, you're not even going to think about some piece of software that your IT people use and how that can really disrupt your entire business or whatever it
is that you do. If you're a government agency or your entire agency to where they basically take themselves offline and investigate what happened and figure out what's going on, and you know, it's just so disruptive to your entire IT organizations, everything else you're trying to do or the IT folks were trying to do gets put on hold until you can fix this problem investigated or do what you
have to do. Yeah, but yeah, that's the I mean, it's the it's a situation that we're in today, and I think it certainly plays into when we think about like DevOps or dev zecops. It certainly has to
be part of what we're doing. Right. So, if you're building software and you're building it on top of open source libraries or connecting to you know, somebody else's API or whatever else, and it's probably several other scenarios, But what are you doing, like what how are you thinking about the security of that piece of software or that service that's being provided, whether it's a SaaS service or what have you. What happens if it's down? That's probably
one of the most most obvious scenarios. But what happens if they they themselves get penetrated by these threat actors? Then what how would you detect it? Do you have the ability to detect it? How would you write? How would you respond? To this, And I think the first step is just sort of thinking through that process, because the truth is is that you can, you know, you can detect these things and you can respond to them,
but not if you're not prepared. Right. It's if you know, the first time you even hear about it is because the FBI is calling you and saying, hey, you've got solar winds and we've tracked that the bad guys are in your environment. It's too late to try to figure out how do we detect it at that point, right, it's now you're just all hands on deck. It just behooves organizations. So really start to think about this, especially with just the number of these situations. Ransomware is. It's
just a perfect example. It's so lucrative. It's not going anywhere, no matter how many times Biden sits down with Putin and talks about how ransomware has got to you know, they've got to put a stop to ransomware. I mean Obama tried to do that with with she you know, back in the day in Shinna several years ago. And you know, it doesn't stop,
and it's not going to stop. So I just think these are examples of that everyone's got to take the heart and figure out, Okay, what is where is the risk to my organization and what are the kinds of things that we can be doing to mitigate that risk. Yeah. I think one of the things you can do that I don't know of a lot of companies that
do this is implement a full on disaster recovery and preparedness plan. You know, I've worked at a few companies in the past where we would you know, we would actually take our backup tapes off site and go to restore them onto this is when we still have physical data centers and restore them onto new
service just to verify that we could bring things back up. And a lot of that practice came from whenever I was in the Navy, because that was a large part of what we did there, Chris, the stakes were a little different, but the principle's the same. You really don't know what the missing pieces are until you go out and try to do that, and that's that's the case. I think. I think that's a situation that a lot of these businesses find themselves in is whenever something happens and they are down,
that's when the conversation starts, Oh what do we do now? Well, the first was to have this conversation before, right before the first step to getting out of a burning building is to plan your exit before the building's on fire. Yeah, it's so true. I mean, quote to that to
that point. How often have you seen people that had an entire backup strategy never tested their backups, have an issue happened right sert of those down or whatever you got to restore to the new system, and then they find out that the backup actually didn't work, or the restore process doesn't work, or whatever they thought was on the tape doesn't exist or right. I mean, it's yeah, it is so common. Yeah, I've seen it a lot and been guilty of it a lot myself. Even you know, even after
like the first time is like, oh that was a painful lesson. I'm gonna I'm gonna learn this time, and now I don't, because every it's always there's always like a unique little best. You know, last time you were missing a certain set of files. You know, this time you had it stored in a location that you couldn't access or you know, so it's like you can't just do it once. You have to do it repeatedly because
the environment's always changing. You know, maybe last time you restored from a particular server that now no longer exists, or you've switched to a different provider
for some particular service. But it's and it's hard. I think it's really hard for IT teams and DevOps teams to get the buy in to spend the time and effort to do that because a lot of the pressure is on build new features, push features to production, increase traffic to the servers, increase the conversion rate, you know, and do disaster recovery planning and execution. Does nothing to increase the revenue of a business. One of my favorite analogies
to that I use it all the time, is interesting. It just resonates well, especially here in America. But you think about football, right, so you know, and it's the same sort of principle like if all your focus is on your offense, because that's what's you know, that's sort of the analogy of the revenue, right, If all your focus is on offense,
because that's where the points could score, and you can't win. If you don't score points, right and you forget about your defense, you're going to be in big trouble because you're most likely not going to be able to score enough points to overcome whatever deficit the lack of a good defense brings you. And then if you think about it, so then you let's say the analogy want to step forward and you say, well, what does a good
defense mean? It means focusing on all the little things, right. A good defense means that if my guys can't consistently do open field tackles, we're going to be in trouble. If I don't have good coverage, we're going to be in trouble. Right, It's the basics. There's nothing fancy, there's nothing like exciting about. You know, you're not scoring a quick six
or anything like that. But if your folks aren't doing that and practicing, and the only way that you are able to consistently do an open field tackle or provide good coverage or all those other little things that you've got to be able to do is if you're practicing them every week. And like you said, you're going back through tapes and like, which is the same idea, right, It's like exercising it and looking at, well, what happened last time? Why did we fail? Or why did this go wrong last time?
You know, you're reviewing tape, reviewing the videos and saying what happened and how do we do better next time? And if you're not doing that, then you know, you think about it a professional football team, like, think about all the support staff that they have on both offense and defense. These coaches, the trainers, the equipment, all of that stuff that goes into both sides of the field. And if you're not doing that,
you're not going to win. And I think the analogy of the business is if you're not focused on both and really working both, you're just putting yourself at huge amounts of risk for something like this to happen, and then you
know you're just going to be in big trouble. Yeah. Response has to be contextual too, because you know, using that analogy, Okay, we need defense to prevent the other team from scoring points, so we're going to send the defensive team out on four wheelers because they'll be more effective that way. It's like, well, wait, no, you can't really do that. And I think the same thing goes with your testing, your disaster,
your preparedness plan. You know, it has to be contextual. It has to be when you propose that, you've got to be mindful of the fact that there is still an offense and you want to minimize the disruption to them, if not completely avoided as well, because I think there have been instances
where the security team is like, oh, that's that's insecure. You know, we have to completely close that off to the point where it starts to the security starts to hamper the business then, and so you really have to find out what that fine line or that balance between the two extremes there is.
I think that's that that's a really important point is that if you if your security team is operating, you know, in a black hole, as the old joke that you know, the security team is always sort of in the dungeon behind locked doors, no one else can get there, and it's that sort of thing. That's that's the worst scenario ever. It really is if your security team is not embedded in the business to where the security folks
understand the business and understands revenue. I mean, because let's face it, what is the biggest risk any business faces the lack of cash flow? It's Microsoft vulnerability, right, It's not the biggest risk you face is a lack of cash flow, right, because that impacts my paycheck. That's everything, right, I mean, that's that's the you know questions is the business open
tomorrow or not? That's that's what defines it. So if the business isn't in line with that, I'm sorry with the security team isn't in line with the business with where revenue comes from. And you know, I couldn't care, Like, if you've got a business process that is where the cashlow is coming from, then it doesn't matter like what the risk of the cyber risk is, Like that's the biggest rish you've got, right, And it's to
your point that that's really what's important. I see that too often that security teams are really just divorced of what's going on in the business. They don't
understand business processes, they don't understand how the business makes revenue. And then those kinds of things that you said, like oh no, sorry, all those ports are turned off, firewalls locked down, and nope, sorry, you know, nobody gets administrative access to X, Y and z, And you know that's what that's when that happens because all we're thinking about is best practice and we're not thinking about risk and what's going on with the business,
and you cannot divorce the deal or else you're just yeah, you're just setting yourself up for even more problems. So so what do you do? I guess if you find that you're a victim of one of these breaches. I mean, we talked about this a little bit with Solar Winds, but I'm just curious to reiterate some of this. It's like, Okay, crap, we got to turn it off before we get stewed into next year or whatever. Right, So that is really probably one of the biggest challenges is what
do we do now? And it depends on what the situation is. So for something like Solar Winds, where they weren't the attack itself wasn't disruptive because thread actor, that wasn't their goal. Their goal wasn't to disrupt people's environments.
It was really to steal information, right, So there the disruption to you is you've got to investigate what happened, because if you find that you are a victim of something like a Solar Winds style attack, you've got to try and figure out what it is that the thread actor accessed, what data that was You may have your own disclosure laws that require you to disclose it. Let's say you found out that they stole information that included personal information or
protected health information or whatever else sort of falls under legal liabilit things. They in Europe you have privacy laws at the GDPR under in California you've got the CCPA, and now other states are following suits. So it's these privacy laws where if sensitive information which is defined under those you know in those laws, gets disclosed by unauthorized parties, which would include you know, one of these threat actors, then you've got to disclose. So you've got to figure that
out. You've got to find out what they did, and that means investigating it, which means having somebody come in and help you with the forensics to figure that out. I mean some companies, bigger companies obviously have big security teams with forensic people on board and all that sort of sort of thing. If you don't, that means you're out there trying to you know, hire inside response firm to do that for you to run that investigation. So that's
you know that that's one possibility. So you know, getting back to like what Will was saying earlier, how do you be best prepared. If you don't have that kind of a sophisticated security team on staff, then you should have a retainer in your back pocket for an instant response team that you know that you can pick up the phone and say, hey guys, I need
your help, like I'm in over my head. You've got to, you know, and you've got to have that practiced and exercised, and obviously the contract signed and all that, all that should have happened before, you know, before you have an incident. But if you if you're the victim of ransomware, in some ways, it's a lot worse because now your entire business or at least parts of your business are disrupted or offline because this ransomware is
on all your systems and you can't access anything. So now you've got to meet this decision of do I pay the ransom which could be very very expensive, and it's questionable about whether they would whether paying the ransomware is going to actually make it any faster for you to recover, or do you just try to recover on your own not pay the ransom, you know, and that's a big decision that you know clearly is going to depend on context and what's
what's been disrupted and what you have in place to be able to recover from. And you've got to answer all those questions. But again it's why you should have something practiced of. Have you be tried how would you recover from, you know, some kind of a disruptive attack like that? Like what happens if like let's say you're a company of a few hundred or few thousand whatever, Well, what happens if seventy five percent of your machines ever and
somewhere on them or fifty percent of your machines twenty five percent? I mean, it's it's going to be a big number. It's not like, you know, for most companies, it doesn't you know, you're not going to be able to just sort of like replace all these machines and get back up and running. It could be a matter of having to like basically wipe and reinstall everything back on a whole bunch of machines, including data. So how
are you going to do that? Do you have the feet on the boots on the ground to bit to do that, or similar to what we talked about earlier, do you need to have a retainer in place with some outside services organization that can help you recover. And this case of this the MSP is what happens if your MSP happens to be the one that gets hit,
then what you know, do what would you do at that point? These are these are tough situations, but I mean as sort of as disruptive as our world has become, these are scenarios that you know, people used to probably call this like one hundred year storm or something, and they're not anymore. You really got to be prepared for these types of storms that can take down your business very very quickly and for prolonged you know, a prolonged period
of time, you know. And then getting back to what we said earlier, your biggest risk is, you know, running out of cash cash flow. What happens if your computing environment is offline for several days? Do you have the cash flow to survive that? You know, you've got to Those are that also enters the equation. So those are the things you've got to
be prepared for. And there are ways to prepare for this, Like it's you know, this is not insurmountable any stretch of the imagination, but unfortunately, most companies that especially the ones you read about in the press are the ones that we're not prepared for it and are really struggling when it happens.
Yeah, I think one thing that you kind of touched on that I want to elaborate on is whenever you call in the forensics team you know, or the experts to help you understand what the impact was, their answer is most likely going to be I have no idea unless you've got logging turned on inside of your system, right because you know that someone got into your network.
But unless you have like audit logging turned on between different devices. I've been into a lot of companies that once you're inside the network, there's no tracking anywhere to determine what you may or may not have done. Yeah, that's a really good point. It really So here's and we could we could spend another entire episode on this, but here in lies I think maybe the next
step. And I'm glad you sort of helped unpack this because I just quickly said, and make sure that you've got some forensics firm on you know, on a retainer. But they're not all created equally. They have different tools that they use, and they have different capabilities and some groups will some teams will just sort of come on site and they're going to use whatever you've got
set up. And if you don't have logging set up, and if you don't have all the tools available for them, you're going to be in a that's gonna be a difficult situation. There are companies though, that what they will do, and so'll come into your environment and the first thing they're going to say is we're going to ship equipment to you and you have to install it. That's step one. Because you don't have what we need to be
able to pull the artifacts that we need to be able to pull. So that's another approach, and depending on again, depending on your situation, that might be the better approach for you. If you don't have a strong security team and you don't have all these tools and processes in place, finding a FORENDS ex firm that will simply bring all their tools with them and and then they'll be able to pull it pull those artifacts themselves might be really necessary for
you. It's it's just one of those things you have to sort of shop for that and know in the back of your mind what it is that you're looking for, right what where, where are your limitations, what are you you know, what is your organization capable of performing? And what are you
really hiring out for. You've got to be really clear on that what exactly are you looking for in terms of that service, clear that you are in that them that they I think that the clearer it will be, like who are the right one or two or three firms that you want to talk to and decide on, you know, which one do you do a retainer with?
And on that subject. Just maybe just one last thought on that is, you know some of the retainers are you know, you you pay up front for them, and then you know that you've got to guaranteed service level agreement that they'll respond and you know, on time for But you might want to have a second one, you know, sort of backup just in case. And there are zero dollar retainers as well, where you just sign the
contract but you don't put any money up front. Obviously you don't get an SLA with that, but it may not be a bad idea to have another one in your back pocket because you just never know, Like this is it's just the nature of the world that we're in right now, is you just don't know I guess. The other thing that I'm wondering because we're talking about Okay, you know, you'll get some forensic team to come in, you know, make sure that you've got things set up so they can get with
the information that they need. I'm assuming you can consult with these companies ahead of time to make sure that you have everything logging what it needs to log. Right. This just seems kind of obvious to me, I guess.
But the other question that I have is I'd like to be proactive and not have this problem in the first place, right, And I recognize that if I'm leaning on another company to provide me a lot of these security services and they get breached kind of like Solar Winds or some of these others, you know, they're performing a function within my network, may or may not be able to mitigate that because you just don't know who's going to get hit next.
My question is what can I be doing, Like, what proactive steps do I need to be taking in order to I guess offset the easy the easy stuff right where some kid with a script he downloaded off of four Chan comes at my postcrist server, right, And the next thing I know is, well, I forgot to update it last week and so now they're in. Right, are there things that I can be doing to mitigate sort of
the script kitty easy? Yeah stuff? Yeah? Absolutely, And you know, I think you know good examples of that is, I mean, how how often you don't even read about it anymore, don't even like it just doesn't even open people's eyes anymore. No one's even surprised when I see it. But if you, you know, go back a couple of years, remember how often we would see cases of like open s three buckets, right, it had all kinds of sense of information or proprietor right information, right,
I mean it's still happening all the time. Or somebody set up like a Mongo dB box and EC two and it's just world you know, just you know, Internet face readable, Yeah exactly, And those are just full of vulnerability. I mean, Mongo dB was never meant to be Internet facing, right that you know, So how do you find that stuff? How do you how do you defend yourself against that? And I think one of the easiest ways, and it's it's not expensive, it's easy to do,
is to have vulnerability scanners. The two biggest product companies out there that do this retenable and qualless, and they both do it from the cloud. You don't even have to like install infrastructure in your environment. And the first thing you do is just have them, you know, subscribe to it not that expensive, and have them scanning your IP addresses, whatever external IP addresses you own, whether it's in the cloud, whether it's in your own data center.
They're scanning it continuously and you're actually reading the report. That step two, you have to lead their work. If you don't, then you're not going to do anything about it. So you know, like if you've got that Postcriss database that's that's unpatched, or this Mango deb that pops up in EC two and it's in it's world Internet facing, the scanner finds it and if you know, and it says, wow, here's all these critical vulnerabilities that you've got. You see that and right away you fix it, right
you fix it before anybody else finds it. So that's a really easy way to deal with It's just continual vulnerability scanning of all of your systems. It's just you know, it's it's that blocking and tackling we were talking about earlier. It's not hard. It's not expensive, but it requires a litle bit of discipline in terms of making sure that you're always scanning all of the IP
addresses that are Internet facing that you own. Ideally, you're even scanning the stuff that's inside of your network because you just don't know, right, You don't just assume that because it's behind the firewall, it is perfectly risk free, because that doesn't exist, right. It's really you should be scanning everything.
But means that you're doing that, and you've got the discipline to make sure that the scanner is operating and it's still running, and you're seeing the reports and you're acting on them, and it's got all of your the latest IP addresses. So when somebody turns up a new VPC in Amazon, that that set you know that that new network that Slash twenty four or whatever it is that you're using is now being scanned as well. Right, that's really
critical. I mean, I would say that's step one absolutely. Step one. Step two is making sure that you've got some kind of more sophisticated anti
virus. I hate that term, but you know, basically there's something more right because we all know how good antivirus is but realian works well after the fact that they add in the signature for whatever it is that you got hit worth, right yeah, but the other stuff still floating around out there, right right, So you have to have something and the idea is that you're also testing it. So and even if that means bringing an the firm and
saying, hey, help me test this. What happens with this actually protect me against ransomware? And if so, like how well does it work? What process would I you know, what happens in one system against ransomware? Is does this protect me against it? Not like proliferating through my network? You know that's sort of going to work these through and do a lot of
testing and exercising. Like we were talking about with the you know, the analogy of the football team, Like it's not like the football team is you know, sitting back binge watching Netflix for six days a week until it's game day. Right. They're practicing, They're watching film, they are busy, they're you know, the most successful ones are are I mean, it's it's
a grouling schedule and that's what your folks got. I mean, they have the grooling schedule, but they need to be practicing and exercising and all that kind of stuff, and if they're not, you're going to get hit by something that surprises you. Yeah, I want to I want to add a little bit to that. When you do the port scanning, I would recommend
aggressively questioning every exposed port. For example, if you do have the postgrass port exposed for your database server, you know, question why because and I've seen this quite a bit in the last couple of years, people who are using database as a service companies like, hey, sign up with our service and we'll give you a hosted Mango database and you don't have to do any of the maintenance on it, and then they give you a publicly exposed URL for it, and it's like, dang, dude, you know that,
like every script kitty on the planet is just beaten on that thing, like a rental car all day long. You know, in your car. There's there's a better way than this, you know, because no matter you can apply to patches within seconds after they come out, but eventually someone's going to get through. Yeah. So, like my golden rule is no port should be exposed except for Port eighty and its only purpose is to redirect traffic to
port or for three for your website and a port for VPN access. And then if you happen to be hosting your own email services, you know have port was it twenty five open to receive email, and that really should be it. Everything else should just be highly scrutinized. And then since we deal a lot with development, one of the other things you can do is a big place for us to get in trouble as developers is unpatched vulnerabilities in the
packages that we use. So you know, if I'm writing code and no JS and I install an NPM package, well that NPM packages built with dependencies on other NPM packages, which is built with dependencies on other NPM packages. So it's almost impossible for me to know what's actually installed on my MPM server.
But in the CICD pipeline, we can install a tool like Snake and it will look through the manifest every time we push code to master and check for vulnerabilities, and then it has a capability of failing to build if there
are if the vulnerabilities exceed whatever you define threshold is. And so that's a really good way to make sure that you are checking and updating your software in an automated process, because if it's a manual process, you're going to get busy, you're going to forget whoever's doing it is going to be on vacation or whatever. You know, there's all these reasons that mano processes fail. But if you can automate that as part of your CICD pipeline, then it
just happens as a course of doing your natural daily activities. I totally agree with that. And just to hit that one home, the Equifax breach back in twenty seventeen, that was that stemmed from an Apache struts vulnerability, and Equifax had patched Apache struts in other applications and they missed it in the one that was compromised. And this all this is part of the congressional hearing. Why I bet somebody feels dumb, right, how did we miss that?
We hadn't solved? How did we miss it? Well? Yeah, and that CEO is no longer there and yeah seriously, But but you know, you use something like Will is talking about, and you're not manually trying to figure out why what was built on a patching struts I don't remember, I mean that was a built twenty years ago. I have no idea, Right,
it doesn't matter. You have an automated process for finding that because all those that equifacturers breached something like two or three months after the patch came out, so there's plenty of time for that to be patched. And that brings me to another point that I want to bring up, and it's a very unpopular opinion, so this is the perfect format to do it. I tell everybody I'm a moron, and here's why. I'm just kidding. I do that all the time. Yeah, I mean it's you know, it's what
being an influencer is all about. Yah. Yeah, you pick the reasons why people hate you. Yeah. But no, like the Equifax breach is a perfect example. You know, they were they were vulnerable for months and compromised millions of people's data and the final fine payout was like what, seventy
five bucks a person. It's like, oh, that's that's appropriate, Which brings me to my whole point of this is, whenever you're using a third party service, you know, a SaaS service, you outsource a part of your business to them, is I think it's really important to question them on what their obligation is when they are breached. You know, it's not if they're going to be breached, it's when they're going to be breached. What
is their obligation to my customers? And in ninety nine point nine percent of all SaaS agreements there is no obligation. So they can leave your customers out high and dry, ruin your business reputation, and if you're lucky, you'll get mentioned on a tweet from the CEO of that company whenever he apologizes publicly
for it. Okay, over, no, it's it's true. And I don't again, you know, that's one where I don't I don't know what the solution is because I've worked with many companies that try to do a good job of their due diligence when they are vetting third parties right, thirty third party relationships, But how do you vet like you know, you're doing business with an Equifax first, they're not going to give you the time of day anyway, because they're much bigger than just met everybody, right, But how
do you know how good their security processes are? How do you know whether they're going to find that Apache struts vulnerability In twenty nine out of thirty of their application servers, right, I don't know. And I think that's that's an example of where you want to make sure that your insurance is covering you because that's a risk you don't want to take on yourself. You can't mitigate it, so you have to transfer it to you transfer it by buying insurance.
And you know, this is I guess out of scope for you know, DevOps folks, But just as a thought, I mean, that's how I mean, there's again, there's a solution to that problem, but it is it's I think that's by making sure that you are insured properly for that risk. Yeah, And I'm not one hundred percent certain it is out of scope for DevOps because I think they DevOps is probably one of the few places in the business that has enough irons in the fire in different camps to be
able to see those bigger picture things. Yeah. That's a good point, you know, because your legal team, your legal team is not going to know who your third party SaaS providers are. No. No, Well, it's interesting that you bring this up though, because, for example, when I'm dealing with like sponsors and stuff for the shows, a lot of times the yeah, they are involved in the process of Okay, you know, here's the contract, here's the here's what we expect, here's what we're going
to get. And I've seen companies do this with their vendors as well, right where they do scrutinize the terms of service, and they they do scrutinize this. But typically it's the larger companies that are going to push for more favorable terms. Right know, you are actually going to help us with these things when they occur. You are going to be involved at this level. It is going to be your fault when it's your fault, right, Yeah,
you know, shared liability agreement start. Yeah, exactly. If you're big enough you can demand those yeah. But if Yeah, for the little guys like you and I, I mean, you're kind of stuck with whatever they're gonna do for you, or go find another vendor that's going to do it for you, which you may or may not be able to find, yeah, or do it yourself, which don't have the resources to either do
or hire to do yep. Yeah. But I mean I think the bottom line though, is that there's there really is a lot that you can be doing to protect yourself. And I want that message to be to resonate that it's not hopeless. This isn't an insurmountable problem. Unfortunately, It's just that many companies are just not spending enough time and focus on the security side and making sure that security is just part and parcel what everybody is doing in their
day to day jobs. It's not just some you know, security team off to the side dealing with it. It's everybody, and you're practicing it, you're exercising it. It's just sort of constant vigilance. And if all that fails, you can always fall back on the Y two K bunker in Idaho. That's right. If I just get a new job, we're in technology, there's other jobs out there, I guess. I guess That's another thing, right, is what if it's not your vendors, right? What if
it's what if it's your coworkers? At what point do you look at it? I'll give you an example. So the company I work for, we have this process where we take the data that we've gathered and there's a group of business folks involved, the QA the data. Right, they make sure that the data makes sense based on what we know about the market and things like that. That we're gathering it from. And then what they were doing it was there wasn't a good interface for managing that, and so they would
actually and they set all this up before I got there. I have to disclaim that because I'm embarrassed by it. But they would export it to an Excel sheet and then mungge it up and then make us check it back into the codebase and run a script on it in order to import it back in. And yeah, I had a fit and put my foot down right when I found out about it. I was like, no, we're not doing this anymore. This was after the cycle had ended. I said We're not
doing it again this way. And I got some looks and I got a little bit of ribbing, a harsh treatment from it. But what point do you look at these situations as the technical person and say this isn't secure, or this isn't the best way to do this, this is a really dumb idea. Yeah, well, I told them I'm not going to be liable
for this data, so you better find another way to do it. And they took me seriously enough to where we're sitting down and actually having a conversation about it now and they're going to need it in like two or three weeks. But yeah, realistically, what if they told me to go jump on a like what can I do? Do? I just do? I quit? Do? I? I work at a company that is large enough to actually have a security team, so I could report it, right, But yeah, I mean, what do you do? And how serious does it
have to be before you go this just really isn't worth it? Yeah, I think there's all of the options are on the table, and it's important before you decide which option is for you to fully understand like the whole scope of the thing. Not saying that you didn't, but like, just as advice to someone listening who says, oh, I'm in this position, have the full conversation, to sit down with whoever you can, which is probably
gonna be multiple people, and say how did it get this way? Because most of the time those types of things, in my experience, have come from just like tribal knowledge, and it's been decades in the process, and
at each step of the way, no one invented this. They only changed one little piece of it, you know, and then over time you've changed enough of the pieces where it no longer resembles the original thing that it was, but since we've all been doing it that way all along, nobody really picked up on that until someone comes in from the outside and gets introduced to it for the first time and they're like, whoa, wait, what is
this? You know, so, I think it's important to have that context of how it got to be that way, and then try to articulate the concerns that you have about it and weigh those concerns and risks against the cost of the cost of rectifying that. And then once you've exhausted all of those options, now you're at the point where you have enough information to make a decision as to whether you report it to the security team or say no, it's this isn't the right place for me and pack your bags. Yeah.
Well, and it's interesting too write because I had the conversation with a number
of people. A couple of the people, yeah, I mean that was effectively their response was, oh, wow, I didn't realize it had gotten that bad, right, and so you know, no, there was no malicious intent or anything, right, But at the same time, it was, yeah, we definitely need to fix that, But nobody was making out a priority until I actually put my foot down either, and so I think there there's some trade offs and some conversations, and obviously it requires some tact
which I do not and never have possessed, but somehow we made it through anyway, and so yeah, I think it's worth pointing out that, Yeah, you have to have the conversations, right And at the end of the day, I put my foot down, and I put my foot down with my boss, you know, who's a dev manager I think is effectively his stated title, and then the project manager. And so they went back to the business people and said, your dev team is not going to move forward
on anything else after a while until this is solved. So if you want to be able to use this process, we've got to come up with a way for you to do it that they can implement for you, because the way we have been doing it isn't going to happen. And right, and so those conversations did happen and it did go the way that it needed to. But yeah, I just want to add to that, Yeah, have
the conversations. I probably could have been a little more tactful in my approach, but at the end of the day, I think at some point you got to put your foot down and just say, look this just this opens us up to all kinds of problems, and we're either going to do this the right way or somebody else is going to be doing it the wrong way, because it's not going to be me. Yeah, but yeah, I
did want to reiterate your previous point. Yeah. A couple of people said, oh, I didn't realize that it had gotten to that point, right, Yeah, And that's just been my experience, you know, is it's somebody creates this thing, sets it loose in the wild. It's like this
old story. I can't remember where it happened, but they put these monkeys in a room with the banana on top of the ladder, and every time a monkey went up to get the banana, they posed it down with a fire hose, and so over time, whenever one of the other monkeys would go up, the other monkeys would drag him back down. Then they started replacing the monkeys one at a time. The time they had replaced all of the monkeys, and all the monkeys knew if anyone goes for the banana,
to drag that monkey off the ladder. Although no one knew why anymore, right, And then the summary to that is that's how corporate policy gets created. Yeah, I mean we are talking to people though within organizations. So how do you start having the conversations about this stuff? Right? Not necessarily the kinds of things that I'm talking about, but maybe more along the lines of setting policies and setting up automations and things like that that you guys have
brought up. If they're not doing it, how do you go to them and say, no, we need to start doing this, or how do you start pushing them to start doing things that they've never done before, or pushing people on your own dang deam. It's tricky, right, because that comes down to like political skills for lack of a better term, you know,
and what type of politician are you? Are you Teddy Roosevelt where you're going in with a stick and beating them into submission, or you you know in Abraham Lincoln that can can vince them with words and you know, sell them on their own virtues. So you've got to know what your own personality
and your own strengths are. But either way, it starts with communication, you know, and highlighting the problem getting getting you have to understand what their perception of that is, and then you have to be able to articulate to them what your perception of the risk is, so that everyone has the common ground. Yeah, that makes sense, and it's a lot easier said than done. I rolled that out in about sixty seconds, but in reality or
in practice, that could take weeks or months. And if you've never done that before, you can expect to fail the first couple of times, which leads to its own set of frustrations, because then you're like, damn, I went and tried what that dude said, and now nobody implemented my solution and they think I'm a jerk. Yeah, I will definitely add to that, though. It does help to know what your strengths are. It sounds like Jeffrey's trying to chime in. So I'm just gonna say what I was
gonna say. I'm kind of a blunt object and I know that, and so I know that my approach at some point, relatively quickly is going to devolve into no, we need to do this, or I'm gonna quit my way or the highway. Hey, it works works a lot. I was going to say that I think it's that's the situation where you sort of realize you'll find out really quickly what the culture is like in your organization. Right.
For instance, like one of the tenets of DevOps, right is the idea of being a learning organization, right, a continually learning at organization. And if that's really the culture, then you if you bring up an issue like this, that's gonna work, right. I mean, you know there's gonna be mechanisms already for you to be able to do that. For the vast majority of us who don't work for organizations like that, it's more of
a challenge. But I think it's also perhaps an opportunity to help your organization and say, listen, here's a here's a problem. And oh, by the way, I'm only finding one problem. I'm sure there's others lurking our culture. Really we should be encouraging people to bring up these kinds of issues and finding better ways of doing things so that we are a so we can
become a learning organization and we can continually do better. Yeah. Well, and it's and for the most part I found that most people if you can, if you can explain why, then most people will at least hear you out, and so it's only come down to we're doing we're not doing this this way, or we are doing it this way, or I'm quitting. It's only come down to that once or twice ever in my fifteen year career.
Right. Most of the time, you give them a good reason and people are going to go, yeah, yeah, we don't we don't want to have that problem, and so they'll right, they'll figure it out, yep. And if it really does come down to that, I also just want to point out that, yeah, you don't have to accept the liability for those issues. You can go find another place to be. Yep. It's one of the fortunate things about working in tech these days is there's a
lot of jobs. Yeah, yep, yep. Absolutely, all right, Well, I think we've kind of exhausted our time, and then some is there anything that we should make sure that we include that we didn't talk about before we're you know, so yeah, I just want to leave something out and then be like, oh, and make sure that you say this when you I don't know, right, This is my secret weapon is every time I bring donuts in, when I know I'm going to have the hard conversation,
I don't know anyway. By the way, donuts really do work there miracle food. But yeah, let's go ahead and do picks then, Jeffrey, do you want to start us off? I had a feeling you were going to say that will Do you want to start us off? Man, I'd love to start us off. I've got this pick today. We're just going to seem out of the ordinary because normally I say really profound stuff, but I'm going to open this one up with As humans, it turns out
we have a dependency on oxygen. And so this book I've been reading is called The Oxygen Advantage by Patrick McEwan. So this is actually pretty cool. I mean it sounds like, based on what I just told you is like, really this is where you're going, But it's actually kind of cool. I'm a certified scuba diver, a certified free diver. I rode twenty four hour mountain bike races for a number of years and competed in that, and
so I spent a lot of time focusing on my breath. And I just got this book a couple of days ago, and I'm just plowing through it because what the guy is going through here is talking about how your body utilizes oxygen. And since we breathe by default kind of out of necessity, we
never really focus on improving our breathing. And because our habits have changed over the last thousands of years where we don't really do a lot of physical labor or not on the move a lot anymore, we actually don't breathe in line with the way that our bodies should be. And as a result, people
who are breathing what turns out to be way too much. And so he's got these exercises in here to help you lower your oxygen intake, increase your oxygen utilization, and increase your carbon dioxide, which improves the efficiency of your muscles. So I think it's been a pretty fascinating read for me. I haven't finished it yet, but I was so excited I wanted to make that
my pick this week. And I think it applies to everyone who whether you're if you're an athlete, or you're doing a lot of physical activity, it'll be specifically beneficial to you, But even if you're not and just thinking that you want to be more active, it's got some tips and tricks in there that will help you focus and refinething there as well. Awesome, I'll have to check that out. Well, I'll talk about on my picks, Jeffrey, do you have some picks? Yeah. So just following up similar to
to Will's. I read a book a while ago called micro Resilience, and it's sort of like the idea behind it is that we sort of understand the idea of macro resilience of having like an exercise routine and that sort of thing that's sort of like long term, like these are the things that sort of keep me in shape and keep me active and keep me healthy and all that
sort of thing. But the idea of micro resilience was the idea that, like, what are the things that we can do sort of during our day that are not necessarily to try and you know, develop muscle tone or aerobic
exercise or something like that. But you know, there's just those moments during the day when we just feel exhausted or we've just feel overwhelmed or whatever it is that just and a couple of things that she brings out in the book is one it's like, you know, sometimes you feel hungry during the day, and her gument is a lot of times it's not that you're hungry.
It's actually that you're thirsty. And many times if you just drink, you know, just ta take some time and like drink down big glass of water or something like that, that it actually sort of re energizes you. And I have personally found that that really helps instead of getting you know, sort of looking for like the quick energy bar or something like that, just drink
some water. It's huge. Or you know, shows talking about like just standing up from your chair and not just like standing up, but actually do some physical activity, like you know, moving your arms around, moving them up up above your head, that sort of thing. Again, just sort of getting your blood flowing, and it just sort of re energizes you, sort of rejuvenates you. Like another example was, you know, sometimes you're on the road. I've been on the road much in the last year and
a half. But if you're on the road, you're staying in a hotel and they have those like little swimming pools. She's like, you know, even if you just like it starts your day off in the morning by just jumping the pool doing a couple laps. Again, it's not for endurance or anything like that, but it just gets your blood flowing and you know,
it sort of helps you just start your day off. So I think there's a lot of really good ideas of just how to sort of deal with those sort of lulls that you get during your day and rather than grabbing, you know, a chocolate bar or an energy buyer or something my dad like, finding other ways that are probably healthier that will help you sort of get your energy levels back off. I love it. And to be perfectly honest, that's one of the things that I've done lately to help mitigate some of the
tension headaches I've had is just drinking water. I don't know what it was, but I cut back on the energy drinks and sodas I was drinking and just started drinking more water. And that's made a huge, huge difference for me. So yeah, yeah, I'm going to throw in a few picks
of my own. So what I was going to say on Wheel's pick about oxygen was just that, and I think I've mentioned it on the show before, but one of my goals is to complete an Iron Man, and so I've been just getting out and swimming and running and biking, and to be
perfectly honest, it's been interesting. I was a swimmer in high school and did a little bit in college, and so you know, kind of my breathing patterns kind of stem from that, even when I'm running your biking, but I have some friends who are who are runners all their lives, right, and then got into triathlons and started swimming, and just the cadence of movement and swimming and stuff has anyway, it has changed the way that they
breed and the way that they exercise and things like that. And so it's been interesting to me to just see how all that comes together. And I'm interested to see within this book as far as picks go. I found so my swim coach, because I'm on a swim team now. In the morning I go to swim practice. I had an equipment issue with some of my fins. Apparently she wanted me to have longer fins, which are more work to swim with but also make it go faster and help keep you on top
of the water. She sent me a link to some fins. I'll put a link in the show notes. But they're kind of a little bit longer, but they're not like the really really long scuba fins. They're more like snorkel fins, and so I'm going to pick that just because they're nice to swim with, but they're not as big time, heavy duty as some of the scuba fins. And then I think I may have mentioned this last week, but I'm still reading Atlas Shrug and I'm really digging it. It's funny.
I picked it on JavaScript Jabber and one of the other hosts basically said, don't take it too seriously, but I find that I agree with a lot of the opinions in there. So I'm going to pick that just because I've really really been enjoying it. And then I've picked up a new book.
I've only read the forward and a little bit of the first chapter, but it's already appealing to me, and it's called The Ruthless Elimination of Hurry by John Mark Comer, and just talks about getting what you want from life and what you can do to eliminate a hurry from your life and some of the stress that you have around some of the stuff that you're probably trying to accomplish with life. So anyway, I'll put links to all of those in
the show notes. Cool good stuff, all right. Well, with that, I guess we'll wrap up this was We've had some really great conversations lately. I've really enjoyed these, and especially just being able to sit and chat and go through some of this stuff related to some of these concerns and breaches has been great as well. So anyway, we'll just wrap up here and until next time, folks max out