How'd you like to listen to dot net rocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot NetRocks dot com. Hello and welcome back to dot net rocks. I'm Carl Franklin and I'm Richard Campbell, and we're here again for your dot net and all things, you know,
geek pleasure. I think so, Uh, Richard, how's it going over there in Vancouver?
Well, if we get our time shifting right and nothing weird happens and the creek don't rise, I should have published episode one thousand of run As Radio yesterday.
Congratulations, thanks man, big milestone a lot of shows. Yeah it is. And do you want to talk about dev intersection? Can we talk about that now? Yeah?
Yeah, we can talk about dev intersection and you know, all all the things come together well, and we're calling it the we're sort of bouncing between the names because there's a lot of emphasis on cloud development, so sort of Azure community development you know stuff as well as there's two other adjacent shows attached to it. But if you know dev inter Section, you know what to expect.
The usual troublemakers including you and me, but also the next Gen AI show, so lots of co piloting and lots of real jenitor of AI, you know, smart folks, the science people so forth, building cool things. And our friend Michelle Busta Monte has started up a cybersecurity conference adjacent to the other two. So yeah, three shows in one. I got news about that one. Yeah, I'm doing two talks at Michelle's security conference. Oh wow.
One of them is secure your Blazer Server Applications nice important, and the other one is a live security this week. Oh cool, I'm gonna do it on stage Channel show. Yeah, we don't actually have the time yet, but we think it might be in the evening, you know sometime.
Yeah, it's a logical time to do it, right. Well, if you working your butt off that week is aren't you doing a session with Maddie as well on Yes Purifying Never Hanks.
I'm really going to town this week. Crazy, I'm not only doing a Blazer workshop one day, but yeah, Maddie Montaguella and I and Jeff Fritz will be there too, are going to aspirify the dot NetRocks dot com website.
Cool live in front.
Of your face.
There's nothing better than a Brownfield action man like actually taking existing code and applying new standards to it and not having a catch fire.
And honest to god, like, I don't really know what to expect because this isn't something that's in my wheelhouse, you know, So.
It's going to be a no, you're on the ride.
Yeah, I'm on the ride. It's going to be a learning experience for me and hopefully, you know, the it will show.
Learn experience for everybody who's there.
It will show in the results. And then a couple of other talks to Yeah, I'm going to be working working hard. All right, let's start off with better no framework, roll the music, all right? What do you got? Do you know what lotty animations are?
No?
I didn't either until today, well until I found this, Yeah, lottifiles dot com. These are free, lightweight animations, but I guess they require a player, right, and so they look like animated gifts to me, and I don't really understand the difference yet. But However, I did find this Lotty player which is in a part of the mud Blazer extra package. So it's a Blazer Lotti player that makes it easy to integrate high quality Lotty animations into your
Blazer applications. And it's a simple new get package and it's you know, very easy to do. I just don't know enough about Lottie to really care about it, but somebody does, because this is, you know, this is kind of an up and coming thing that's interesting. Yeah.
Cool, another format.
Another format, Yeah, exactly. So that's what I got today. Know it, learn it, love it. Who's talking to us?
Richard grabbed a commenta of a show nineteen fifty five which you did at Build not that long ago, of course, published in June, somewhat after the show, where we talked to our friend Nicole Forrestkrenn, who hadn't seen in a while,
one of the you know, Devop's goddesses. She ran Dora for the longest time and really was the person, I think, more than anyone, who put the numbers to what it meant to be a high performing team, like what the difference was when you organized into rapid iterating ways, integrated to and so Forth said, here's the performance different differences that these teams get across the large numbers of organizations.
So she kind of took the you know, anecdotal side of DevOps away and gave it a real science rigger. And we've talked about this concept of frictionless development, about what happens when you can code quickly and have this infrastructure around you to know what's going on, and that included using the contemporary AI technologies, although that was you know, in May, and here we are in August September, where of course things have.
Changed again light years away.
Yeah, it keeps moving so fast. And so this comment comes from Neil Tiberwalla who says, thanks for the great episode. On one of the projects I'm working on, we are also struggling to see how llms fit into DevOps. However, are surprised to hear you all talking about multiple agents merge conflicts. It seems like this is really a non issue since the AI could just delete their work and start over and redo the whole task. It's just some
computation not but otherwise not lost human programmer time. That is the motivation for a merged process. Yeah, there's also a cost in tokens. But I'm with you. Yeah, And as you said on the show, a new paradigm requires new ways of thinking. Definitely, software che all that is changing again, and I'm not unhappy about it. It's interesting to talk to folks and to experience all the different approaches. So Neil, thank you so much for your comment. And a copy of music Coby is on its way to you.
And if you'd like a copy of music Koba, I write a comment on the website at don Arocks dot com or on the facebooks. We publish every show there, and if you comment there and I read it on the show, I'll send you a copy of music Goba.
Music to code by I remember that, Yeah, that thing I started a long time ago to help you focus during development. Well now there's twenty two tracks, and if you don't want to send in a comment and get a free copy of it, you can go to music too coode by dot net and you can download the entire collection in MP three, flack or wave format.
Nice.
All right, So this being showed nineteen one hundred and sixty, let's talk about what happened in nineteen sixty six. There's you know this is one of those years, right that people remember space exploration, Soviet Lunar nine spacecraft, and I'm sure we'll talk more about that. The nineteen sixty six FIFA World Cup held in England. The host nation won its first championship by defeating West Germany in the final. The Cultural Revolution in China Mao Jeidong right, so significant
social and political upheaval. The Vietnam War. The US announced a substantial increase in troops in Vietnam. Civil rights Sammy Young, junior civil rights activist, was murdered in Alabama, highlighting the ongoing struggle for civil rights. The New York City Transits Strike, a wildcat strike by public transportation workers, began January second and lasted until January thirteenth. But my favorite category, music and culture. The Beach Boys Pet Sounds was released, which
totally reshaped pop music. John Lennon sparked outrage with his comment that the band was more popular than Jesus.
Yeah not smart.
I shouldn't have said it, and I'm sorry. And The flint Stones aired its series finale on April first, And there's of course a lot more of it. What do you want to tell us about technology?
In space Richard, You know, ninety six six is not a huge here on the space side, just because it's the end of the Gemini program and just before the Apollo program really ramps up. Although there's lots of Cold War stuff going on, lots of reconnaissance satellites and so forth. But I would point out that Surveyor won launches and lands on the Moon successfully for the first time, one of the very first soft landings on the Moon ever.
Wow.
And also they start doing mapping of the Moon with the prospect for landing with an orbit called the lou Or orbiter. And it's a time before CCDs and so forth, so taking pictures and sending them back is not a simple thing in nineteen sixty six, So yeah, it's not the most exciting year sings. They'll get hairy from here
as the Apollo program really gets rocking along. But at this but on that year they still did a few important things and it wouldn't be You're not wrong that the Soviets had the Luna twelve mission, which also took photos of the Moon, although admittedly after the Americans got them. But let's talk about important stuff, like nineteen sixty six is the year the cool whip is invented, all right, nothing like an edible oil product.
Yeah, edible oil. Cardiovascular dream come true.
Listen, palm oil is well, let's be clear, not your friend.
Not your friend.
On the computer side, Helid Packard's very first computer, the HP twenty one sixteen A, which is a funny number for your first one. It was actually an acquired computer. They bought a comp called Data Systems who made a thing called the DSi one thousand, although they had never shipped. They bought the company back in sixty four finished it up.
It was all integrated circuits, which is very novel of the time, and magnetic core memory because they're still a couple of years away from digital ram being a thing. This is a sixteen bit mini computer with a ten megahertz clock. It comes with four K words sixteen bit words, but you could externally expand it that's more coarse up to eight k and if you wanted to go to a big old sixteen you need an external box for it.
Wow.
Weighed about two hundred and thirty pounds and cost about twenty thousand dollars when it first shipped in nineteen sixty six. And finally, yeah, very relevant for the time. In nineteen sixty six, a scientist be the name of Joseph Wisenbaum working at MIT, released a piece of software called Eliza.
That was so fascinating to me.
There Rogerian chatter bought, as they called it at the time, ran on an IBM seventy foot ninety four. Whisenbaum talked about it doing language transformations, which is interesting to think about sixty years later.
All right, so the real story is that the term artificial intelligence was being bounced around, and he knew that these were just programs, and to prove it, he wanted to say, you know, this might seem like a human, but it's just looking for keywords and spinning out wrote answers and actually not even answers, answering questions with.
Questions questions, the Rogerian approach.
Yeah, yeah, so if you say, you know, I hate my brother or whatever, it would say, tell me more about your family. But he did it as a goof really, yes, he did. He did it to show how ridiculous it was, how ridiculous.
And people spent hours with it.
It's true. Yeah, I have witnessed it.
Yeah, they did run it through a Turing test. And it failed. Of course, not that the Turing test is particularly meaningful, but this is in the early days of it, but it was one of the they later on to come on to define this idea called the Eliza effect, which is this a tendency for humans to overvalue language interactions as especially intelligent answer promatively. You know, language is a pretty profound thing in this sort of sentient space, right, so we're kind of wired to respond to that for
better or worse. But hi, golly, aren't we battling these same problems right now?
Oh, one hundred thousand times more.
Yeah. Now we call it chat GPT psychosis.
It really is too, isn't it. I mean, it's very serious. I'm not going to say who it is, but somebody that you and I both know what. You know. He was trying to talk to his mother about AI or whatever, and to impress her he basically spit out a document. He went to chat GPT and spit out a document, you know, in minutes, and she was amazed by it. And I said to him, I said, did you check the facts? Heause, no, no, I didn't. It was actually
an application. It was like something that you would send to somebody to apply for something, right, and it had all sorts of facts and did you check it? Because you know, the thing does get stuff wrong. He goes fabricate stuff whole cloth.
Yeah, and he.
Didn't even think about it.
Right.
It's like, you know, and this is an intelligent because the computer is always right, right. This is an intelligent person that you and I both know, and you know, just forgetting to fact check. Yeah, yeah, yep, scary all right. End of Soapbox nineteen sixty six, you know, sixty years ago. All right, Shall we introduce our guest, Michael Levan. Michael
translates technical complexity into practical value. He's a seasoned engineer, consultant, trainer, and content creator in the Kubernetes, security, DevOps and platform engineering space, spending his time working with startups and enterprises
around the globe. Michael is also a Microsoft MVP in the Azure space, in AWS community builder three times, a four times published author, a podcast host, an international public speaker, CNCF ambassador, and he was part of the Kubernetes version one point two eight and version one point three to one release team. Welcome to the.
Show, Michael, Thank you so much for having me now, I have the friendly reminder that I need to cut down my bio a little bit.
Actually it's quite impressive and I wouldn't leave anything out.
And he's knocked out a couple of run answers with me over the years too.
Yep. Yeah, but your first time on this show.
Yeah, yeah, thank you guys so much for having me really appreciate it and worries.
Well, you know, part of DevOps is dev so oddly enough, you do you do have your hand in that space as well?
Yeah, no, absolutely, I mean, ironically enough, I feel like I kind of live in two worlds. I do a lot of you know, the devopsy cloud based stuff, but I would say probably as a consultant, half of my a year or so is spent doing a lot of back end software engineering. So I've been doing a lot of Python lately. Before that, I was doing a lot of Go in my free time, you know, outside of client work. I'm diving deeper and deeper into Rust. So I'm definitely in all the worlds ironically enough, so.
All the worlds. Yeah, it's good to be a polyglot, right, to exercise the different languages and sort of see how the various sides live, although I just went through a process where we took a sample ap of an MVP and regenerated in six languages in a matter of days. It's like, here's what it looks like in Swift, Here's what it looks like in Rust. Like, I mean, you're don't even wonder what language even means anymore.
The irony of it is, I've been thinking about this a lot, is that I wonder if we'll reach a point where like natural language will be the language.
Yeah, you know.
I wonder if we'll hit a time where it's like, yeah, you know, I want to write a memory, you know, sensitive application, and then you have two choices. You say, well, you can go with you know, this language, which is maybe Go, or you go with this language, which is maybe you Russ. And then you say, okay, this is what I wanted to do, and then it goes and it does it. I don't know, maybe we reach a point where it's like, you know, everything's just bytecode and
we just kind of don't see it. It's all natural language.
Yeah, well, if you never look at the code, what do you care what it was generated?
Unless it's job ASCRIPT, then I would protest, Yeah, do care? Hardily.
You know, mostly it's about understanding what's going on. I was thinking about cucumber the other day. Remember this testing lie where you're supposed to sort of write natural language tests, and it's kind of you know, we've always tried, you know, what is specifications or requirements, but natural language that describes code, that is ultimately our measure for how we provided a solution which it seemed to be moving further up there.
And it's not like any of this stuff doesn't take a lot of skill to do.
Well.
Yeah, you know, it's you know, this is not a trivial practice, but certainly evolving without it.
So Michael has DevOps I don't know changed in the last three years. For you, what's been the most significant change in DevOps in the last few years.
I think, you know, the obvious answer is AI, right, But I think conceptually, like I always look at it in two different ways. I look at it in the positions and like what you're doing in your day to day at your job. And then I also look at it from a like underlying technology perspective. And I always say this to everybody, like, if you have good fundamental knowledge, like you know, if you have a decent fundamental knowledge and you know, the computer science and stuff, right, so
like data structures, standard algorithms, all that fun stuff. And then if you have a good base of you know, standard IT networking systems, how all that works. The irony is ninety percent of what you're doing is kind of staying the same, and you're just sprinkling different abstractions and different tools on top of it. I often think about like service measures as well in the realm of kubernetes.
You know, they bring your network observability, your traffic routing, your MTLs, encryption between service and service, all that stuff we had already. We just now have this layer of abstraction that this program that sits on top of a service versus embedding you know, TLS code within your application code. So point being is I think a lot of it
is still the same. I think what's just changing is like the different tools that we're putting top on top of Okay, cloud based environment systems all that.
Certainly the human aspect of DevOps, which is you know, people, processes and tools. I guess the thing that Richard likes to talk about all the time, the people stuff. Has that changed because of AI or are the people skills still there and required.
Oh one hundred percent still required? Totally, totally. I think that with bringing an AI into DevOps and anything right like software engineering, maybe we could say security as well, but I don't think it's it's there yet, but a lot of it is around, like DevOps and software engineering. And in my opinion, I think I need to work harder now using an LLM uh than I did before, and I think that the level of effort is just
changed a little bit as the human operator. So like before, for example, maybe I'm sitting there looking at a line of code for an hour saying to myself, like what is going on here? And then I finally found it and I say, hooray, I have fixed all my life's problems. Now now I'm.
Where can I get me some of that?
Now I'm generating, you know, this part of my application with an LM. But now I'm looking through the whole thing and I'm saying, this isn't right. I'm going to change this, this isn't right. I had a scenario the other day where I wanted to build on an MVP of something and I built it and I used claud code and I was running it, and I'm like, why am I result? Why are my results the same across every system that I test this on? And I looked at it turns out Claude cod I didn't ask it
to do this. By the way, claud Code created a bunch of mock data within it and with a bunch of print statements, so like I literally had to go in and refactor everything because it was just all fake data that was getting printed out to me on the terminal. So like I had to spend probably more time fixing that than I would if I just wrote it myself. So yes, I think the Humanator, the Humanator, the human.
That was my nickname in high school.
It's nice.
I think that piece is actually probably more important now than ever, to be honest.
Yeah, And unfortunately this is another one of those shows where we're talking about AI. But you know that we catch some flak for that and our listeners like it when we just get back to real development. But but you can't deny that. You know things are changing, and this is one of those things we've been saying this a bunch that you have to you now have to think of yourself if you're going to embrace these tools.
You have to think of yourself as sort of a manager of lower level engineers and fact checker, right or or code checker, you know, code reviewer. If that's the role that you're taking, you know, and hearing we have a situation where it's even taking longer because of that than if you did it yourself.
Yeah, I mean, and we aren't you know, like having these layer of layers of abstraction isn't new, right, Like in the beginning, if you wanted to test something out, you had to write your own compiler and you had to write your own editor just to run this thing. And now I pop open VS code and you know, I run my high level programming language and I and
it's good to go. So like there was a time I'm sure where everybody was like, oh, we're gonna humans are going to go away, and we're going to lose all of our knowledge because you know, we're not writing our own editors to run this application anymore. So it's we've seen this countless at times, you know, through the last sixty five years of software engineer.
Also, I think that there's a an expertise that gradually grows when you're interacting with these things so that you know how to interact with them. The problem is that they change so quickly and change so often that it's hard to know. You know. We we live with a single compiler, a single ide for years and we finally get it figured out, and then they make a switch, and then we have to figure something out. But it
takes years. Now these things, who knows, they could have changed in the back while you were taking a nap or drinking coffee, and and the way that you interact with it to be most productive has changed. Totally awesome.
Yeah, yeah, but you know, it's all part of the evolution of things too.
Right.
I'm I got concerns. I do have problems with the hype cycle. But what I like about DevOps is it is in the product, right. It is a set of disciplines that make a better team. It's another one of those terms that's kind of fading away in the sense that this is just what a high functioning team looks like. That everybody's pulling towards the same goal of delivering solutions to customers that we care about, both its creation and its deployment and its operations and the telemetry and how
that feeds back to making a better product. The fact that the tooling shuffles around underneath us par for the course, right, you could hope these things are only going to get better.
Yeah, And I think you know to that point, a couple of years ago, five six years ago, maybe if you went and you wanted to do a talk on DevOps at a conference, I mean, that was the thing, right, like every conference shows, right, Yeah, that was it.
That was it.
Now you know it's different because to your point, Richard, it's just all kind of there in the background. It's just everybody's just kind of doing the thing.
Uh.
You know, we we're we're seeing it very similar with Kubernetes. Like Kubernetes from a container orchestration perspective, that's just the thing that you run it on. It's not it's not the hot sexy topic anymore, but the stuff that you're building on top of it is.
Uh.
And I'm sure you know I will go through that same thing. The one thing that I will say though, is when you're not programming day to day, you get rusty quick.
Oh sure.
So if you're constantly using you know, whatever your favorite m is to generate your code for you, you may be able to like, of course, still look at it, you know, figure figure out this is breaking here. I got to put some air handling over there. Blah blah blah. But like if you're just staring at a blank ide, you may have some trouble after a while, like getting something out on paper, so to speak.
Sure humans are better at editing than they are to creating out a whole cloth anyway, yea, And Heaven help you if you try and build a CICD pipeline from scratch and YAML on a blank screen. I'm sorry. I cut and paste it all and then I go through it and try and understand what I just paste it in and make the tweaks accordingly, like we build on the shoulders of giants.
It's the same for writing pros or poetry. Is that totally You don't write, you brainstorm and then edit. Writing isn't about writing, it's about editing.
Yea, and all these things are the same. I'm just realizing we talked about in the earlier versions of Visual Studio how hard it was to build out the tooling, all the bits and pieces you need to actually automate the deployment pipeline. Today it's almost like we have an abundance of riches, like you're gonna get hub action this as you're devopsing this, yes or terrorforming this, like you have a lot of choices.
I remember when customers would hire app Phenex to create a CICD pipeline as a you know, a consultancy because they didn't really know how to do it. And now it's just click click click boom boom.
Well, and it was kind of a one off thing too, Like why would you want to get good at this? Although let's be clear you should, although today, like you said, it's just a set of checkboxes, Like the real crazy part is that you didn't. I still feel like the automated tests aren't great, although I gotta say looking at the prompt magicians, the guys that are really good at using these tools, they they are generating tests as part of the code and iterating on the test, passing with
the code using the tools. Like arguably, if you do a good job with this, like you're doing that, you're you're insisting that these tools do the ideal case you never even measured up to yourself.
It is very true.
There's nothing like nothing like one hundred percent code coverage when you don't have to do.
It and it's all about the templates, right. It's like, you know, do I have to write my one hundred thousandth unit test or mock test or integration test or functional test. No, it would be really great if I could just offload that to my AI friend, whichever one I decided to go with.
You know, the one that I really struggle with is good telemetry. It's easy to collect a lot of data, it's hard to collect the data that really tells you what's going right and what's going wrong m hm.
And particularly know how to make sense of it.
Yeah, we're clearly getting some sense of messages coming back here. We have reams and reams of data coming in and we're paying for all that. But are we learning anything about how people are using our app or are they what their frustrations are?
Like?
I think this takes a lot of thought.
Yeah. Are you still seeing the same problems in twenty twenty five with DevOps that you were three or four years ago? Yeah?
I mean I feel like I'm seeing the same problems that I saw, you know, fifteen years ago. I think so many of the the fundamental problems don't change all that much because the fundamental problems are honestly typically people problems, right It's you know, I think that's just how it always is, which is why I you know, a lot of people are nervous right now because you know, they think, hey, I'm going to take jobs in this now, blah blah blah.
But that fundamental thing doesn't change, you know, until the terminator like universe is fully in play here. I think we're all all right. But you know, in terms of like DevOps as a whole, I think the biggest things honestly right now, phinops is a huge one that everybody's trying to figure out, like the whole idea around cost
optimization and with that performance optimization and resource optimization. Things are moving, I mean, and I know we always say this, right, but it feels more so now that things are moving, you know, quote faster than ever. And with that, there's just four more resources being used. And now we're seeing a lot of of like actual like cost optimization or finops engineers or whatever the title is getting hired in organizations specifically to do this job. And that's very interesting.
So I'm definitely seeing a lot of that. But in terms of you know, the general what DevOps is doing. I don't think much of it has changed recently. I think the I don't know if you would call this a DevOps change per se, but the whole idea around platform engineering, I think really what they're trying to do with that is, say, I have engineers that are managing and building and maintaining this particular tool that we're utilizing
internally like a product. Right, So they're putting their customer service hats on because you know, their internal engineers are the customers of the platform. You know, they're building it in a way where they're thinking more about architecture than they are about let me go write the code, and then they're building this thing for use to be able to move faster inside. Now, we've always kind of had tools like that, or you know, rather engineers that would
build stuff like that. You know, when I had a senior Principal Infrastructure engineer title years ago, when I was building out little automation tools in Python, which would maybe look like you know, platform engineering today. But I would say that's probably the biggest shift that I've seen recently in this space overall.
Sure, it seems like a good place to take a break, So we'll be right back after these very important messages stick around. Did you know that you can work with AWS directly from your ide AWS provides toolkits for visual studio, visual Studio, code, and jet brains rider Learn more at AWS dot Amazon dot com, slash net slash tools. Now we're back. It's dot NetRocks. I'm Carl Franklin. That's my friend Richard Campbell, Hey, and this is our friend Michael Levan,
And we're talking about DevOps in twenty twenty five. Because you know, some things have changed, but some things have not.
Uh. Security, I'm just going to drop that bomb out there, because yeah, you know it's the battle a. It's way more difficult these days. The black hat's gotten way smarter. Oh god, it's got to be part of the equation. A lot of this generated software people are not putting enough consideration in. They're much less. Not that they did but when they were writing it all by hand. But how do you see a fitting in the pipeline, Like what's the culture like when to make this work? Well, I like to.
Look at it from the software side and then from the networking side. So from a software from an application perspective, how you're securing your application, whether you're going with you know, your standard app set stuff like your SaaS stuff and your SCA stuff, what libraries you're using, you know, are you encrypting traffic? What does the code quality actually look like? So that's one piece of it. But from a DevOps perspective, ironically enough, I feel like the majority of security issues
are around networking. So you know, if there are any network admins or network engineers listening, you know, the whole idea around what does ingress look like, what does egress look like? How are services talking to each other? So like, for example, in the realm of Kubernetes, are you using a service mesh? Are you encrypting the traffic going back and forth at the L seven layer? Are you doing pod to pod encryption? Are you, you know, encrypting at
the L three and the L four layer within your cluster? So, and that's just an example, of course, but like I think that's honestly, in my opinion, the biggest part of it is what traffic is coming in, what traffic is leaving, and perhaps even most importantly in the world of containerization, how are these containers, are these pods talking to each other, and how does the traffic look. Is this pods supposed
to be talking to that pod or that container? Are these services supposed to be calling out to this database? And I would say, honestly, that's a majority of the security issues in the realm of debops and platform engineering. The other big thing is and this is I think just across any realm of it. I think one of the red hat reports that I read maybe last year or the year before, it was like seventy four seventy
six percent of security issues are due to misconfigurations. Yea, and that's just everywhere.
And then I'm patch servers, you know, throwing on my run ass hat, like for honest the goodness, I think we're actually making headway on on SQL injection these days, so that it's I think it's a third now. It used to be first forever. But it's like, you know, misconfiguration. You haven't secured the thing properly in the first place, and you didn't get the patch donet a time. Like we've had huge discussions on run ass about what's higher risk, deploying a bad patch or not deploying the patch as
quickly as possible. And it seems like that sort of flip that I'd rather the outage because I deployed the patch fat than the exploit that comes from not getting that patch out there quickly, which is crazy, but it seems to be the new reality.
So it seems to me that you've got two sort of realms of security when it comes to DevOps. You have the configuration of the topography itself, and once that's done and everything is secure and working, now you have a continuous job, which is you need a software bill of materials. You need to know what devices you have, what their versions are, and update them every chance you get. So you need to keep on top of that, and on top of that the software, the libraries and dependencies
that you're on. You need the graph of dependencies so that and you have to watch the news. So if there's an exploit in one of those things, you need to know that, and you need to you know, if it's hardware, you either have to patch it right away. Sometimes the only thing you can do is take it off the network because there's no patch available and you're vulnerable.
Worst case scenario. Yeah yeah, but hopefully you're going to security Microsoft and you're watching the CVE stream, like exactly, that's probably the first place to look, Michael, are you big on things like the actual API management? Like that saved my bacon a couple of times now just using those tools, Yeah, to show I know who uses my APIs,
I can set thresholds per user. So when somebody got exploited and started doing mass extraction of data, the API limits kicked in and kicked up a bunch of warnings. You know, they had legit credentials. Everything looked fun. It's just the traffic was out of shape because it was an exploit.
Yeah, one hundred percent. And I think tools like that we all need to be looking at constantly because the problem is, well it's not even a problem, it's just something that maybe we all still have to get used to, is that there are way more endpoints now and way
more layers to networking than there were before. You know, to take because we're talking about DevOps, right, take the Kubernetes example again, Like there was a time where you just had this monolithic app and like there was a server and there was one entry point, right, and everything was there and it was all good. Now they're just multiple layers, like you know, you're looking at the Kubernetes cluster. You have your host networking layers like your virtual machines
and stuff and everything running it. And then you have the container networks, the pod networks. That's another layer that you have to manage. Then you have your Kubernetes services, which are another networking layer, and that's where you know your security centric cni's and your service match like histio and stuff will come.
To Gee, I wonder why we have configurations problem, Michael, I'm confused. I'm on my seventeenth layer and I feel really good.
Well, and that's that's honestly the Again, it's not a problem. It's just something that we all have to really get used to because as where you can call it decoupled applications or your micro services or whatever whatever phrasing you want to call it, everything is split up now and everything is taught to each other. It's not just in one box anymore. And because of that, you now have not only multiple layers, but a lot of east west traffic,
a lot of north south traffic. Like everything is talking to everything inside of a cluster, but then also outside. Maybe it's hitting a public endpoint, maybe it's hitting a database. There's a lot of traffic going back and forth, and that's why I think, you know, outside of the software engineering security stuff, I certainly hope AppSec gets more and more popular, especially with AI right now. It needs to. But aside from that, from a purely DevOps perspective, and
I think the majority of it is networking. Like you have your patches, of course, and you want to be able to update your APIs like from a you know, let's say like you're utilizing a specific API within an application running. You're not managing that application, but you want to be able to upgrade those versions. That's one thing. But yeah, I mean I'm going off on a tangent here, but a big part of it is networking, honestly.
Yeah, I arties like why haven't I set up an agent that's watching the CVE stream from Microsoft and just evaluating against every project that I've got they exist, Yeah, I appresume they do. It's this is non dirigional line idea.
Well, the MCP stuff, but it's.
A great idea. M M Yeah, you don't necessarily want to give it the ability to update your firmware, but no, but certainly I just want to heads up right a heads up because would be good. Yeah, just reading all those things like reading a CVE is painful. What right, They're not fun to read. But before you can get that list effectively, you need to have software building.
Materials are at the other side of this, right, Yeah.
So are you seeing s BOMs slowly creeping into the culture or still no? I mean last time I checked that there's not not everybody's doing those. Yeah.
I don't think as much like it. It hasn't grown in the same way as like your general security practices, like for example, like policy as code, you know, making sure that you're following all the best practices and you're using something like open Policy Agent or or you know, Kaiburno or one of the other policy as code tools. I think stuff like that we're seeing more so, but things like s BOM, things like you know, your your overarching cybersecurity pieces. I don't you know what it is.
I just don't think it's like it's not like a hot topic, you know what I mean.
It is it's very preventative, but like it.
Doesn't sound good, you know what I mean, And people are like, ah, you know, we don't have to worry about that. Because it's not you know, a top five thing in the realm of cloud native or whatever, and that's just that those are the hype cycles and all that fun stuff.
So well, sbom got more love when log for Jay got exploited. How bit exposed are we? Right? Like how many of these things only get dealt with after something's on fire?
Right, And that's the general security unfortunately, is that it's really important when things go bad and richer to your point of like you know, hey, I want to set up an agent to be able to listen for, you know, any TVs that are coming in or whatever. Then you then you got another security issue, right, is what the agent telling you accurate?
Right?
Yeah? Even correct?
And then.
Yeah, around and around. Well, but this is you know, who puts on the tinfoil hat in your organization?
Right?
Like where We're lucky, there's somebody wears it all the time and they like it, which is rare. Uh, And that's a strange person. And I you know, I'm very aware that there in a few places like Okay, one some month, I'm going to put the hat on and like today I'm the security guy, I look a little more nervous and I'm angry all the time, you know, but you know, and you put your foot often you find yourself focusing on preventative work, right, and maybe that's the time when you build that thing.
The tinfoil hat guys are generally focused on conspiracies, but fortunately for them, you know, hacks and exploits are conspiracies against you.
Is it still a conspiracy if it's really happening, Because you're looking at the logs and just watching them hammer away at you, it's like, is this a conspiracy or it's just a log file?
Like, I'm pretty sure the tinfoil hat guys think people are inherently evil and then they get proof of it and it makes them happy to reinforces their tinfoil hatness.
Yeah, it just but if nobody's looking, then you find out the hard way.
Yeah sure, yeah, Like I'm greeing with you, Richard. I think you got to put those guys to work.
And I've included a link to the gethup repository of the s bomb tools that Microsoft put together.
Cool.
Oddly enough, I think it was around the time of CrowdStrike. I don't know one of those you know, conspiracy another one gees, how could that have happened. It was earlier than that. Maybe it was it was one of the other like supply chain acts, like there's been a few. It's just you know, it's frightening. But it's of course
this is not actually a security show. This is just how do we include this in our overall life cycle of making software that it does get going And you know, again it's in some ways it's like we're going to be the better version of ourselves because we're writing this as a prompt for tools, for these tools to actually try and implement it right up until it gets hard and it's like, no, you actually now have to work on this just because you found a CV that might
be relevant and it relates to a UNK code that is deployed in your organization. Now you got to figure out what to do.
Yeah, And I think a big thing is it's really hard for organizations, specifically management to like specify an ROI on we're going to take the week or whatever it'll take to make this application a little bit more secure, and no, sorry, we have something something deadlined. Sorry, something something needs to be out this week at the end of the quarter.
When you're also avoiding the big thing, which is that the consequences to the company getting exploited never seem to be that significant either. You know, you if you engage your PR crisis team, you do your make help us on TV if you if you're a large enough scover that thing. All of the customers that got exploited, you sign them up to some kind of monitor during service and say sorry, and then you go on with your day.
Yeah.
And I've even heard, like quite literally, I've heard we don't want to know.
Yeah, yeah, because he represents liability.
Yeah. If I don't know, if you knew, you know, it's gonna blame me. Yeah.
Yeah, it's so. And I'm not a big on willful ignorance, but holy man, right, like come on, yeah, and again it so the side effective the run last time of dealing with real security guys where quit a job because they refused to deal with it and then when it went down big time, they got sa panted.
Right right, yep. Yeah, it's not a uh, it's actually not an uncommon thing. I mean it's very Yeah. Security is securing your application in general, securing your your DevOps piece of the puzzle. I mean, it's for for everybody listening. If you're wondering like, oh, do I have to go and dive into security full time? Or you know, what
do I gotta do here? Really, it's just when you're designing your your pipeline, or you're designing your Kubernetus cluster, you're designing your application stack, whatever it is, just take the extra five to ten minutes to think it through, you know, and a lot of it again, it's it's so many just simple misconfigurations. Oopsies. I left root access on these r back permissions when I was testing something
and I forgot they were there. I mean that's probably like in the top three things that happened, right, you know, Yeah.
Yeah, deployed with it with the dbug still on, and people can stap in back black hat stap in it and take advantage of it. Liketh, it's it, Yeah, well were I'm not clever enough to think of these things. All I have is the case studies.
Yeah, it's really, honestly, so much of it, especially especially from a DevOps perspective, is just yeah, take the extra you know, breathing through your nose and out through your mouth and just say okay, does does the look okay? And honestly, if you if you just take that couple of minutes, I can guarantee that you're gonna find some stuff. And you're like, oh, this would have been bad.
Yeah, well that's the standard cycle of development too. You know, when I'm developing, I always when I'm done, okay, there, I get up and I walk away for fifteen to twenty minutes, and I come back and I go through it again and just you know, take a deep breath to think. But you know, now now you have to think about it from the perspective of security, you know, and everything else, not just development.
And that's that's tough for a lot of teams, right, especially for DevOps teams, because honestly, I think that the biggest problem is I've spoken to a lot of people, and a lot of people are like, yeah, I wish we could do this, but we don't have the time. Yeah I wish we could do this, but we got to get this done tomorrow. Yeh, I wish we could. And that that's it's always what it is. So sometimes you have to manage up as well. Again, and it's
it's never a technology problem. It's it's always a people thing, not even a problem, right, It's just you need to be able to and this is going to be different
across every organization. You need to be able to think about what your manager, how your manager receives information, and mold what you want to tell them in a way that gets them to understand, ooh, this is important, we need the extra day because usually a manager can find the extra day or the extra couple of hours as long as you present the information in a way that
they can digest it and it's important to them. If you're frantically running around and saying, oh, this thing needs to be done because of this and that and technical jargon here and technical jargon there, chances are a manager's going to say, yep, just another person that you know just wants extra time to do this thing and to put, you know, a couple of extra sprinkles on top. I've
seen this a thousand times in my career. But if you're the person that goes to your manager and explains it in a way that makes them realize just how important it is, and hopefully you know your manager well enough to know how to do that, If you don't learn, then your outcome is going to be very different. So really, I mean implementing all the security implementations can be as simple as just understanding what the other person needs to be able to hear to make it an effective decision.
You know, one thing we haven't talked about is social engineering from outside forces into you know, the staff on the edge of the of the organization, you know, clicking on emails and all of that stuff, and then somebody getting control of your network inside. I mean, the way to prepare for that isn't necessarily with technology, but education of your staff. And I wonder how how often that creeps up in your line of work.
I would say not as much as like, if you're on the help desk or you're you know, in the cisadmin space or something right now, you're probably going to see that a lot more. The only time that I personally see it is like if I accidentally click on a phishing scan, which I'm human like, sometimes things look legit, you know, and I'm like, oh, this was one of those no before tests or something like that.
But I guess, like the standard backup practices protect against ransomware, right, so if something does happen, what do you do? And you could spend an ordinate amount of money on protecting yourself with multiple backups and off site and all of this stuff. How do you how do you prepare for the or do you even prepare for ransomware? Are you just like, now, if that happens, we're screud.
Should you prepare for it? Absolutely? Like taking the backup example. Last week, I actually with a client, I did a dr strategy test where we were on the call for about three hours and we blew a database away and we tested to make sure that things were going to work as expected to luckily you know we you know, they test recoveries. They don't just back things up and hope for the best.
Now, you don't have backups unless you test.
Them exactly, that's it. So things like that, I mean, yeah, like you can absolutely sit there and prepare for But again, it's one of those things where you want to do it, and you can do it from a technology perspective. It is there and it's readily available, but now you have to go in convince your manager that you need to take to three to four hours to go and test this thing and confirm that it's going to work as expected. Ironically enough, in the realm of DevOps and in the
realm of a lot of engineering. When you get to the senior at to principal level and you're the one that's supposed to be making these decisions and calling these things out, your job is a lot more sales than it is engineering. You have to go and sell this idea and you can do it, and the technology is there and there is trust me, there are twenty vendors as we speak that want to sell you their solution that can do this thing, and they can do it
very very well. Now you've got to go sell that idea to upper management.
Well, the best way to do that is put a dollar figure around it, Like if we got ransomwared, what would it be worth to you to get all this data back? You know? And they're going to know that. So ten million dollars, twenty million, thirty, one hundred million dollars like those aren't out of the out of the realm of the ransoms that we've seen paid totally. And so you convince the management with dollar figures, what is the risk if we don't do this? Yeah?
A thousand percent?
Always yeah, And it's always good to bring this numbers to the CFO and bring scary headlines to the CEO.
That's it.
Yep, yep, yeah, and you just you really just got to be able to sell the idea of like what can happen. And for your manager it could be those
dollar figures. For another manager, it could be, well, if this happens, we're going to have to hire a consulting firm and we're going to pay them five hundred thousand dollars to do this, Or could be we need well, we're going to have compliance issues, right, and we're gonna you know, we're going to mess up and we're not going to be able to get our High Trust serve or our SoC two compliance or whatever it is. So you just got to figure out what's important.
Could end up in an audit, you know, if your financial services he gets hairy.
And it's something as simple as how much would it cost us per day to not have our data exactly right? To be offline in one day would cost x amount? Right, And now you're so you're more likely to pay the ransom to get your data back.
Accept it. Often when you pay the ransom, you don't get your data back.
Yeah, that's true. You know, there's no guaranteed failure rate is.
Failure rate's pretty high you know, we have taken that story where we're a certain security person actually had to fix the decryptor when decrypt files larger than two gigs. Yeah, and the reality and we've done this show and run as as well. It's a year minimum. You know, you're going to have people involved, You're going to be cleaning up messes like it's just nothing's fast. It takes a long time. And it takes the same amount of time where you paid, where you paid the ransom or you did.
I've heard of companies doing testing their staff by sending out phishing emails, you know, and seeing if they actually click on them, and you know, when they get there, you say, congratulations, you just ransom.
You just signed up for additional training.
Yeah, exactly, And a lot of them.
Are actually pretty good. Like you know, the old method, for example, used to be, you know, hover over the email and make sure that the domain doesn't have like a zero.
Instead of an ozer exactly.
And now it's you know, everything's legit and you're like, yep, this is a legit email and you click it and it's like oop, sorry, so.
Yeah now and the well, now you get into the other out level of this, which is that be more careful next time? Is not a strategy now that we actually have got a.
Lot Another big one is you know, here, copy this file off my USB stick right oops. So in the studio here, I have customers that come and go with their own data all the time, and I have a machine that is not on the Internet, that that has you know, malware bytes and all that stuff on it. So I get the latest updates, I unplug it from the Internet, stick the thing and run a scan on it, and if it's okay, I feel comfortable enough that I can put it in my real computer, right right. Yeah.
So cynical, Yeah, I wish I didn't have to be, but yeah yeah.
But is this the pain point these days? Really just think it's the security side of things when it comes to software, Like I feel like a lot of the a lot of stuff has been well automated deployments automated now like that part works, Continuous deployment like that, that seems to work pretty well. If you put in the time, you can get your results. Total testing still seems to be a strugg but that's evolving. But actually having a sense of the security quality of your app, I don't
have good measures man like that. You only find that out in the field, it seems.
Yeah, And I think to that point, the biggest thing right now, going back to what I was saying before, is the networking aspect, like where are my packets going? There's so many different layers now, yeah, east west traffic, north south traffic, so many different layers that it's like, where is this going? And do I know where it's going and what it's doing and who can hit it?
Did it really need to go there right right doing unnecessary things?
And I guarantee, I guarantee that can fix ninety percent of the security issues that you see in like a devop slash platform engineering environment right now where there's a lot of you know, Kubernetes and there's a lot of cloud.
But it speaks to whitelisting, allow no traffic except the paths I've specified, and I don't see very many folks working.
That way totally the defense and depth, but you know, it's it's all that stuff. It's least privilege right, like block everything and then open up as necessary. The problem is again that's that's that takes a lot of time and people are like, hey, I got you know something
something deadline, that's it. That's it, yep, yep, yeah, and it's it's it's wild to see even even like let's say you're setting up different metrics toolings, right like maybe you're you're setting up Jaeger, you're setting up Prometheus, and you're you're collecting you know, end to end app metrics, you're collecting standard server metrics. Whatever you're collecting. Go look at those sometime and see what's talking to what from a networking perspective, and your mind is going to be blown.
You know, it's not not even controlling it, just monitoring it.
You just look at it.
You'll be surprised even.
Like every once in a while, you know, I'll I'll open up this this fancy new tool called wire shark. Yeah crazy, and I'll look at what's talking to what, and I'm like, I don't even know.
What this is.
Although I'm minutely wire sharks better at marking updata now than it's ever been before. The problem with wire sharks you turned that on one. Okay, there's a lot of stuff. I have no idea what it means. It's overwhelming.
Yeah, yeah, it's all fun.
Until somebody loses an eye exactly, sure enough. So Michael, what's in your inbox? What's coming up for you? What's next?
So really still focused in the Kubernetes realm, I'm going to be focusing more and more on the networking aspects of things, Securing networks, observing networks, making sure that proper traffic routing is going where it's supposed to go. Is this thing is supposed to be talking to this thing? And then also in the realm of AI, you know, looking at various agents, how are these agents talking to
one another? How are you talking to agents? Really everything around the Kubernetes networking security realm, and then of course just overall platform engineering, making sure that what tools are implemented are necessary, which I call in my head just proper architecture. I think what's next for me is just proper architecture.
That's good, good, Okay, Well, thanks for hanging out with us for an hour.
We learned a lot.
I know I did anyway, and thanks again.
Thank you so much for having me.
You bet, and we'll talk to you next time on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios Service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at PWOP dot com. Visit our website at d O T N E t R O c k S dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September two thousand
and two. And make sure you check out our sponsors. They keep us in business. Now go write some code. See you next time. You got javans
And