Is AI ready for DevOps?

it has so much potential that. In the first quarter of 2025, I kind of thought this was gonna be a big deal. This was gonna be, uh, if we can figure out how to keep these things from hallucinating and going crazy in our infrastructure, this could potentially be the AI shift for infrastructure that I was waiting for. So started this podcast. We recorded our first episode at KubeCon at the beginning of April, 2025, and this is gonna be a series of very specific episodes around getting.

Ais to do useful automation and work for DevOps, platform engineering, infrastructure management, cloud, you know, all those things beyond just writing YAML, right? So the, the intro for this podcast, there's a separate episode for intro. It kind of goes into my whole theory of why I think this is gonna be a thing. And this episode we really try to break down the basics and fundamentals for those of you that are catching up. Because it's a lot. There's a lot going on.

It seems like We have announcements every day this year around AI agents or Agentic, ai, however you wanna call it. I am calling it Agentic DevOps, and hoping that name will stick. Now, this episode, since it's from the beginning of April. And it is technically now just getting released at the beginning of June. We're a little bit behind on launching this new podcast. Um, I think everything in it's still relevant. There's just been a lot more since. And I don't know the frequency yet.

I don't know how often this podcast is gonna happen. It could be potentially every other week. It could be weekly. I just don't know yet because we are not gonna do the same thing here as on my usual podcast. If you're someone who knows that one DevOps and Docker talk that I've been doing the last seven years, that one is still gonna have AI in it.

But this one is very specific and there might be a few episodes that have syndication or whatever you wanna call it, of the episodes on both podcasts. But most of the time we're gonna keep the focus of just everything, DevOps, everything, containers on the DevOps and Docker talk show. And this one is gonna be very specific around implementing useful AI related things for Agentic DevOps, or automating our DevOps with robots. So I hope you enjoyed this episode with Nirmal from KubeCon London.

Hey, I'm Bret. And we're at Kon. We are Hi, Nirmal.

Yeah. How is that different?

Terraform in it or whatever, and I'll just copy those commands and check 'em and write them myself. So the LLM is not executing anything on my behalf. On, on your behalf. Agent would be defining a tool set. So I could give, I could define a tool called Terraform or a tool called Shell I could describe what that tool does in natural language.

Generates the terraform just like it did the first time around. It knows what command to run. It generates the command and then the magic here, a little box will show up and says, do you want me to execute this on your behalf? You click the button, you click the button, and then it executes that Terraform apply Uhhuh and it sounds very simple, but it's a very different paradigm in terms of thinking about how we interact with infrastructure or systems in general. Like broadly systems in general.

Because we are no, like in this way of looking at it or thinking about it, I, as the human, are no longer executing those commands. I am. Trusting to a certain extent that the LLM can figure out what it needs to do and giving it a guardrail set of tools to use and execute.

I now think I've got a much better handle on this. I've been spending so much of my life right now, deep diving into this, to the point that you and I are talking about changing some of the focus this year on, on all these topics. Absolutely. Because I think this is gonna dominate the conversation. This is, these are, there's gonna be a lot of predictions in this and we're not gonna talk forever 'cause it's gonna need to be multiple episodes to really break down what's going on here.

But we now have the definitions. AI agents, what are tools? The protocol behind it is essentially MCP right now. Although that's not necessarily gonna be the only thing, it's just the thing right now that we're agreeing on by one company. Exactly.

That was the moment of 2013 and that, and yeah, I'm not alone in that feeling. yes.

and right systems operations, it's still in that same vein, but. Yeah, we're not replacing,

And so we are just still using human effort to do that work. But this, I feel like I'm gonna draw the line in the sand and say, this is the. month or the definitely the year, that kicks off. What will be a multi-year effort of figuring out how we use automated LLMs Essentially with access to all the tools we want to give it with the proper permissions and only the permissions we want to give it right to do our work for us. In a less chaos mon monkey way, right? Like less chaotic way.

Potentially. Potentially. It could, this thing can easily go off the rails. Absolutely. I will probably reference in the show notes Solomon Hike's recent talks about how they're now using Dagger, which is primarily A-C-I-C-D pipeline tool. So he's talking, and a lot of my language is actually from him iterating on his idea of what this might look like when we're throwing a bunch of crazy hallucinating AI into what we consider a deterministic world. Correct.

One is the validator, one is the tester. One is one is the builder. And they can fight amongst each other. And it all works out. It actually ha happens to actually work out better. And so if you're like me and for the last three years of understanding, ever since GPT. 3.5 or whatever came out.

We all saw chat GPT as a product, and then we started with GoodHub copilot and we started down this road As a DevOps person, I haven't had a lot to talk about because I'm not interested in which model is the fastest or the most accurate. 'cause you know what? they all hallucinate and still even today, years later. Code agents and we and you can see this on YouTube, you watch basically thousands of videos on YouTube of people trying to use these models to write perfect code and they just don't.

And so we in ops, but we look at that, I think, and the people I talk to even for years now are like, we're never gonna use that for ops. But now my opinion has changed. Yeah.

You're in the right place because that's exactly what we're gonna explore.

Yeah. and we're gonna explore with you where that, goes. Bret (2): Yeah. This, just to be clear, this is KubeCon April, 2025 and almost no one is talking about this yet. It feels like it's right under the surface of a lot of conversations and a lot of people maybe are thinking about it, but I'm not even sure that we're honest with ourselves around. That this is coming, whether we like it or not. And only because, yeah, not only, but one of the large reasons is business. Okay. Lemme back up.

You know how in a lot of organizations, Kubernetes became a mandate, right? So there's lots of stories that came out over the course of Kubernetes lifetime of teams being told that they need to implement Kubernetes. It didn't come from a systems engineering approach of solving a known problem. It came down. Because an executive decided that they read a CIO magazine article that said Kubernetes was a cool new thing and they did it right. I hear this all the time.

I confirm this multiple times this week with other people, and I now feel like we're not talking about it yet. But I did hear multiple analysts say their organizations that they're working with expect that we are going to reduce the number of personnel in infrastructure. Because of ai. the only way that's possible is if we use agents to our advantage, because we can't, yeah. I still don't believe we're replacing ourselves. I don't think the agents will ever in, in the near term.

And as far as we can see out, let's say five years they will, they won't be running all infrastructure in the world by themselves. They can't turn on servers.

They maybe you can actually pixie boot and do a power on a POE or whatever, but. Like we still need someone to give them orders and rules and guidelines to go do the work, but to me, I'm starting to wonder if very quickly, especially for those bleeding organizations that are looking to squeeze out every cost optimization they can of their staff, that they're going to be mandated to not just take AI as a code gen for yaml, but to start using these agents to.

Increase the velocity of their work . And my, one of my stories is over the last 30 years I do this in talks is every major shift has been about speed, cost reduction in speed. Sometimes we get 'em both at the same time. Sometimes they're one or the other. We get a cost reduction, but we don't go any faster, which is fine, or we're going faster, but it's not necessarily cheaper yet. Right.

To automate more of these platform designs, platform build outs, troubleshooting when we're in production and things are problematic and we don't wanna spend three hours trying to find the source of the problem. If we're not starting to use agents to, to automate a lot of that and reduce the time to market, so to speak, for a certain feature or platform feature then I don't think these teams are gonna hire more of us to help enable the devs to deploy.

What it could end up happening is we end up more with more shadow ops, where the developers are so fed up with us not speeding up to the, if they're gonna go 10 x we have to go 10 x. Yeah. If they're gonna go three x or whatever the number ends up being in the reports. And Gartner puts out like the AI makes it efficient, more efficient for developers to, to code with ai. And the models get better and the way they use it is better.

And so they're shipping code faster and they can do the same speed with three times less developers, or they can just. Produce three times more work, which I think is more likely, because if it's the common denominator and everyone has it, then that means every company can execute faster and they're gonna, they're gonna want to do that because their competitors are doing that. So that's a's, that's a very loaded and long prediction.

An organization can't adopt AI until the humans learn how to adopt AI and the humans have to go at human speed. So we can't just flip a switch and suddenly AI is here and running everything for us. At least not until we have Iron Man's Jarvis. Or whatever. Like until we have that, we still have to learn these tools and still have to adapt our platforms to use them. Yes. And adapt our learning to use them. And that's gonna take some time

This agentic kind of thinking Around even just executing the agentic platform, if you will, And everything from looking at the tokens and optimizing load balancing to inference endpoints or MCP is, doesn't behave the same way as like just an http connection. Necessarily. And solo. We were talking to them and they have an MCP gateway. We're seeing a little bit more of a trend on AI gateways.

Is DO the project has an AI gateway and so this is not just another workload and looks like just a web server. And the networking and everything is gonna be different. Not dramatically different, but We'll, but drift different enough that we need to be aware. 'cause even if you're not using any of these tools, someone in your organization is probably gonna say, oh, we need to integrate this stuff into our software, to our right. Whatever we're delivering.

And we'll need to know it even at that layer. So we're gonna also cover that component as it relates to. The Kubernetes ecosystem, right? And cloud native.

here's my Kubernetes API, here's all my other things that I, you have access to, and here's my problem. Go solve it. And that paradigm. Three months ago, two months ago for me, I didn't know existed. And that's why I've been sitting on the sidelines with ai. Like it's cool for writing programs that mostly work in a demo. It's cool for adding a feature to something I already have, but it's not doing my job as a platform engineer or DevOps engineer.

It's just helping me write text faster Then I can type into my keyboard. And that was not that interesting. That's why you didn't see a lot of me talking about that on this show, was it just wasn't that interesting. This is an interesting topic for ops and for absolutely engineers on the platform.