Charity Majors on AI, Observability, and the Future of Software

It's just, you know, he's like, and the only stage that remains is monitoring and monitoring needs to evolve because it's actually the substrate that all the stages lead up to that, right? It is the validation, it is the source of truth. It is the the, you know, source of meaning. So over the past few days, like, I love Boris is also, full disclosure, like, a friend. A he founded Baseline, which is another observability two point o startup.

And I think by the time this podcast cast comes out, he will have announced he's doing a new thing with agentic AI. So Boris and I tend to have very, very simpatico, like, philosophies and approaches and everything. And when I think back to, like, what inspired what drove us to sound Honeycomb. Right? It came out of our experience at Parse.

And Parse was a lot like Heroku, mobile back end business service back in the early twenty tens. And we had over a million mobile apps running on Parse at the time I left. Facebook shut it down after the acquisition. I will never forget them. Assholes, they didn't have to do that, but they didn't anyway, this is not but one of the challenges of operating software at pars was running a million of labs.

We never get to talk to those developers. We don't know what they intended. We have to understand that code from first principles. We have to operate it from first principles. And I feel like right now, I am shocked that everyone is so shocked that they have to understand lines of code from first principles.

I'm like, oh, that that is not a that is not a transition that everyone went through. It's something that we went through at PARS, and it seeded Honeycomb. But the other thing I wanted to say was, I think it was just yesterday, maybe it was Sunday, my friend David Pole, p o l l, he's now senior director of engineering at GitHub. He was also at Parse. He posted this great piece about how, I think the title is, code review was never about finding bugs.

Right? Because, like, one of the things that Boris was arguing in his piece was that code review, it's antiquated, like get rid of it, it's just a, it's point of friction, it's not adding any value, know, and I'm like, yeah, preach it brother. Dave was like, wait, That's not what Cutter U is supposed to be about. David was on the so I was on the infrastructure side at Parse. David was on the APIs team, right?

And David and that whole team would spend hours debating whether or not a PR because it was about the mental model. Right? It's about do we let this into our product or not? Can people reason about this? Does this make sense? Is it coherent? It's like that. Look. Code review needs to evolve. No questions.

Right? Like, we're we've been overloading code review. He and one of the my favorite lines from his piece was something about, I think the word code in the word in in code review gives people the wrong assumptions or preconceptions. It's not about the code. It's about the product. Right? Yes. Which as I said that, I was like, oh, yeah. That makes a lot of sense. That makes a lot of sense. You know, we're we're using code review to do so many jobs.

And then, what is so fascinating is that I looked at that in the experience and I just laughed. I was like, yeah, but can it scale? Can it is it is it can you operate it? Can it run billions of you? I just laugh, I was like, right? No, there's still a job for ZOB. AI is great at generating, it brings it up or down to the medium. Right? That's what it does. It doesn't doesn't not reasoning.

It's bringing it to the most common average level. So if you're below average, cool. But like the more you know about a domain, you apply AI to it, the more you're like, oh, this is this is not good. Right? But my friend David Powell would have looked at that same Slack ex you know, generation thing and said something like, yeah.

Of course, they can vibe code up a clone of Slack, but the hard part of building Slack was never adding features with a with a you know? It's the reason Slack is so cool is because of all the Mhmm. All the design decisions, all, like, years of compounding decisions about how to make this intuitive, friendly, all of these things. That's the word. And so I think it's I know I've been, like, seven minutes straight, I've been rambling about this. But, like, I think that's so shab both sides. Right?

It is far less clear to me that this will work for all types of code. Right? Yeah. And and I think that from both extremes, both the reliability, the dependability extreme like, a a clip that I have been saying for at least fifteen years is the closer you get to laying bits down on disk, the more paranoid you have to be. Operating systems, databases, devices, you and it's and a point that a front end friend made of mine is that it's actually not about disc on disc, it's about dependability.

It's about dependency stacks. Right? How many things depend on you being predictable, and resilient, and not changing too fast. Right? I think users too, from the other angle.

Nobody wants to use a Slack that is subtly changing from day to day. You know? And and I think that, like, the great frontier that I haven't heard many people really intelligently speak to yet is, obviously, disposable code is great for rapid iterations and trying things and blah blah blah blah blah. But the way that we know how to build maintainable, reliable software that people love is by taking what is known to be good and iterating on it in small, small frequent changes. So how do we cross from one to the other?

That's an open question, because I don't know. But like, it's very clear to me that that's a really big question. And where does the human where are humans need it? I mean, it's very easy for machines to generate code. In some ways, you could almost just squint and be like, this is very similar to the advent of compilers.

Right? We don't even think about what's There are like 20 people in the world who think about what's happening when C code meets hardware. Right? It's still really important, but it's not something everyone has to think about or care about. Same goes for infrastructure. You know, when I I am just old enough that my first job had the title of sissabin. I was one of the last sissabins. Right? And then we got into this whole like, oh, ops is a synonym for toil. Fuck ops, you know, blah blah blah.

And fuck those guys, first of all. Turns out, operational excellence is one of the lingering differ differentiators, I think.

Speaking to ops, infra, DevOps, engineers, being like, I really, this is me speaking now, but I feel like you could be very cynical about AI, yeah, just swap until sometime in 2025. Now it's clear that it is going to be better than the media and engineer at generating logic, if it's But one of my favorite Adam Jacob quotes is something about, we are actually very fortunate that this is a technology problem. Unlike manufacturing, technology problems require technologists. So his advice to everyone is, don't take refuge in being cynical, right? Lean the fuck in.

Learn it, level up, learn its rough edges, learn to be literate, and, you know, you can be still be cynical, but your critique isn't gonna land, or nobody should listen to you if you're just, like, ruling it out and not touching it. Right? So it is a technology problem. Check out our need technologists, but it we need to earn it.

They might be right, but they are just speculating. Right? I am a year ago, I was giving the closing keynote with my coworker Fred Haber at SRECon, and our big argument to SREs for why they should learn AI was so that your critique will land. Basically, can slow things down better if you pretend to like it. You know?

And at some point in the last three to nine months, I realized that that's actually not I don't think that's a legitimate critic anymore. I mean, I don't remember, I don't know who said this, but you know, when the facts change, I change my mind. What do you do, sir? You know? Yeah. I don't think I'm the only one who's had that experience in the past year.

Do canary testing, automate Railmax. You know, look at your fucking data after you ship it. Is it doing what you meant it to do? Does it is it doing anything else that looks weird? Like, close the loop.

This is the fastest, easiest, best way to find problems before you like, and I've come to think of that as eat your vegetables lecture. And one of the questions I asked myself was, why is it that these this I hate the term thought leadership, but whatever. Why is it that our, quote unquote, thought leadership has been so much more successful than our product? Why is it that people don't go, yeah, that's what I want, and immediately go sign up for Honeycomb because the opinions built the product and the product built the opinions are the same thing. Right?

And what I've realized is to eat your vegetables lecture, it what? What percentage of teens have all those tools and get to work that way in fast feedback loops? 158%? I've never heard a credible estimate in the double digits. It's the 1%. Right? And other people might read those and go, oh, yeah. That would be great. I want to go to there. Right?

Or maybe I'll put one of these things. It's not within reach or they'd be doing it. They're not stupid. Some of the best engineers in the world work at these companies with two months, six month deploy pipelines. Right? Because it's harder. It's actually harder to work that way. Requires more of you. And what I've come to think about AI is this is this is our chance to bring our engineering values to the mainstream. Mainstream.

To make it accessible. A a quote that I found myself using a lot lately is that AI is like alcohol. It is both the cause of and solution to all of life's problems. Right? And it oh, right? Well, I got

And it's to us. It is fast to it's fast for humans. It's a lot to better be for humans. AI is 10 x, 100 x, thousand x faster. And our systems are held together at this rate of change through a variety of stupid human tricks. Right? Intuition, the person who's been there for fifteen years. The documentation that's out of date or isn't written down, but someone knows where to look, right? And when that rate of change tips up, all the duct tape flies off the bus.

It is virtually impossible if your company has ever dipped below a certain line of technical debt. You can't climb out of it. You can't pay off that debt fast enough to get to get your groove back on a capitalist, like, quarterly timeline. So anyway, this is how I found my way into being like, this is these are the engineering values we have always cared about. And this is our way to take them mainstream.

Hear that from FREs. I hear that from, like, front end platform engineers. They're like, they're like, oh, when we were just doing this for our engineers, people weren't willing to fund it or prioritize it. Because it was always seen as infrastructure, not value add. Right?

Hate for their output to be measured in any way. Others, this whole like, DX and Swarmina, and even I would like to feel like engineers at Honeycomb have a fair amount of emotional safety. There's everywhere I see a lot of, well, it doesn't work because blah blah blah blah blah blah blah blah blah. You know? And while it is absolutely true that there are different ways of creating value, capturing value.

And it is absolutely true that many companies have misused these metrics. And I get why there are scars. We have to be willing to look at our output, and talk about it, and be like, in the language of the business, and the lowest common denominator is time and money, right? Which we can convert to some extent one into the other. Engineers don't need to worry about their skills not being useful, but it's the nature of technology that's the way those they constantly need to be changed, evolve, mature.

And more than ever, I think we need to get comfortable with talking and thinking about them in terms of the business.

What I think the first one, I'm probably gonna say some bud buzzwords here and I apologize in advance. But I think that your if someone had to say an example of a DevTools founder who's like created a huge personal brand and like leveraged that to like, many buzzwords, to grow their business. I feel like you're probably one of, if not the best example. Oh, been you been able to I think people get stuck on I've seen this. I do it myself.

Like, it's so easy to get stuck on, like, just you're you're, you know, you're a CTO of a very difficult comp like, know, a hard engineering problem. Right? Like, be very easy to just spend all your time just on internal stuff, I would imagine. How have you also been able to, you know, create all this content and and really put out such a big message at the same time, which is the sort of thing that usually might be someone that was a dedicated marketer and probably would still struggle to have that kind of an output.

And this is honestly, this probably would not have played out this way, but I was CEO for the first three or four years. I was CEO for the first three or four years. Our entire strategy our entire go to market strategy consisted of any place that invites Charity to speak, she says yes. So I was flying all over, constantly jet lagged, constantly on Twitter, and, you know, Emily Nakashima was our first front end engineer, and she stepped into that position. She's still here.

I couldn't do it. I couldn't do what I do without her, and like, our partnership just being and the engineers, we they're just they care about the craft. They care about the culture. They care about our users. We've always selected for very, like, pragmatic business, customer focused engineers who are not no one at Honeycomb is like, ew, marketing, or ew, sales.

Know, do you understand they're there to build a business? And I respect them deeply. So, like, I am not in engineering day to day, and I always forget that people probably assume that I am. There's no way you can do both. You just can't.

If you're curious, if you're humble, if you talk to people who do know their shit, you know? And I very much found this to be true. I find out what I think by writing about it. When you look at it, it's external. Is that there's a hole in that argument.

That doesn't make sense. I need to ask someone about this because this doesn't you know, it's a way of making sure that what you believe is consistent and grounded. So, in fact, funny story, not so funny, it's been a miserable winter. In working on the second edition of the book, at first I didn't really want to write it, because I was like, I've been writing about observability for almost ten years. Do is there anything left to say?

And I should know this by now, but of course there is. Oh my god. I learned so much about cost structure and POCs. And so the first if I can detour just a little bit and talk about my own problems. Absolutely.

So we started writing the second edition of the book. We got to get my co authors and I got together last June, and we planned out a new table of contents. And I pitched my co authors. I'm like, I think there should be a brand new part six at the very end. We'll call it observability governance.

And it'll be a bunch of topics that observability engineering teams want me to know about. Because observability engineering teams didn't exist when we wrote the first version of the book. Observability didn't really exist. So over the summer, June, July, August, September, I said I was gonna write four or five chapters. I was in the last one in September, and it's about buying software.

And I was like, you know what, I've never actually bought software, I should probably. So I put out a column in my blog, I'm like, if you've ever bought software, send me an email, give me your advice, what do you wish you had known, right? And oh boy.

And for like six weeks, I couldn't I'm just get up every morning. I've got eight hours blocked out. I'm just maybe I'm tinkering and then I revert. Just like, just wasn't coming. Then the week before Thanksgiving, I was at midnight fucking around with Claude just like and I realized, oh, shit. Everything I've written needs to be thrown out. I I wrote the wrong thing. I wrote the wrong things for the wrong people. Do observability engineering teams need help? Of course they do.

Who doesn't? You know? But but the stories that I got from people, again and again and again, like in text and subtext, they were describing a technical decision maker buying process from the top 10. Was completely messed up. Just like no alignment between CTOs, VPs, SVPs, distinguished engineers, principal engineers, staff engineers, directors, managers, like, no.

They're not even trying to get consensus around what observability is, what problems because it's changed so much, so dramatically. And everybody been their eyes have been fixed on the AI ball. Right? So, like, one of the questions that comes up is, is this even still relevant? Isn't this just an IT ops problem?

Isn't this solved? Didn't Datadog do it all already? You know, they haven't updated their priors in years. And DPs and CTOs are the worst, because typically, they haven't really been hands on, even if they think they are. They haven't really been hands on for ten, fifteen years.

And so, they still are technical enough to think they're on top of shit, but they have the opinions that they formed when they were an engineer in the trenches like a decade and a half ago. And so, they're just like, I got this. They don't. They don't got it. So I threw out all the chapters I had written and drafted a new list of chapters where it starts with a open letter to CTOs about how everything that they wanna do over the next few years is probably blocked now on organizational learning speed.

The ability not to generate code, but to understand it and make decisions based on their understanding it. And what do we call that? Observability. You know? That's what observability is.

And then because every term in the industry is so used and overused and copied and sales and marketing and so we just don't even use them. Like, we just talk about it from a systems perspective, like fast feedback loops. Right? There are two kinds of feedback loops. There are balancing feedback loops that cause stability, and there are amplifying feedback loops that cause change.

But what creates a feedback loop is observability. If you don't have observability, you don't get a loop. You just get shit happening all the time. Right? And and if that Yeah.

That observability is lossy, laggy, wrong, nonexistent, you're flooring it with point proof. I mean, like, it's just not gonna work. So, anyway, I we our last chapters just went into tech review, I think, today, maybe. And I'm really looking forward to the second edition coming out, but, that rewrite almost killed me because I was just like, I'm supposed to be done by now.

I'm not which is not to say that you shouldn't write about your work. You should write about it. More people should write about their work and really commit to that because doing your work in public, being willing to be wrong in public and be open about that, I think it builds credibility with anyone, but especially engineers. It holds you to a higher standard because, again, writing is thinking on paper. It is a way to accelerate working out the kinks.

You don't have to build out every problem to make sure that it's worth doing or not. Sometimes you can talk and write your way through it. I've given a lot of talks over the years, and I don't regret it. I, believe it or not, I when I started giving talks, 02/1132, I could not think and talk at the same time. I got such powerful stage fright that my legs would wobble and lock up so I'd be terrified of tripping.

I just I had nightmares for weeks before I gave my first talk. And it is worth building that skill set if you're a founder or a leader of any or, like, if you want it's worth building that skill set, and it is eminently buildable for anyone. But I think that the impact you leave on the industry is largely through your writing.

I spun up a Substack over the holiday break. It's charity.wtf.substack.com. I'm on Twitter, Mipsy Tipsy, LinkedIn, ugh, all of the social medias. The book, the first 20 chapters are already in prerelease on O'Reilly's site. The rest should be there next week or two, and we're gonna be putting out some preview chapters on the Honeycomb website between now. The book, the Dead Tree version comes out in June, so between then and now, we'll trickle them out.