Kubernetes and retiring at the top with Kelsey Hightower

And how he kept creating new opportunities for himself, often unknowingly. The inside story of the container wars, Puppet, Docker, Terraform, CoreOS, and how Kubernetes eventually won. Going from a Google IC to executive level, and how he rejected a Microsoft offer. from Satchio Nadella himself and still doubled his compensation. His grounded and pragmatic advice for software engineers worried about being commoditized by AI and so much more.

If you're an engineer thinking about your long-term career trajectory, whether that's getting into staff plus level, going independent, or even quietly planning to leave the industry, This episode is for you. This episode is longer than a normal episode, frankly, because I was so glued to my chair, mostly listening to Kelsey's stories and things.

This episode is presented by Antithesis. Verify your system's correctness without human review or traditional integration tests and avoid bugs or outages. Before we start, I'd like to mention our presenting sponsor, Antithesis, and maybe offer a little history. Over the last two decades, software development has gone through a mindset shift from an imperative approach

to declarative one. Infrastructure is a perfect example. Think about how tools like Puppet and Ansible allow declaring how individual servers should be configured. Then came Terraform, the ability to declare the desired end state of your whole infrastructure. servers, networks, databases, and their relationship. And then with Kubernetes, we stop scripting container lifestyle.

Instead, we write manifests that say things like, I want the replicas of this application exposed behind a surface with this much CPU and memory. Once we didn't have to specify every little detail of our infrastructure anymore, deploying software became much faster. But then the bottleneck became how quickly we could test and verify the code to be deployed. Testing remained imperative. We had to write tests for every little detail.

And now with LLMs, we're on the verge of a declarative shift in the way code is written as well. Just tell the model what you want and let it figure out the details. And it's going to make the verification bottleneck a million times worse. Anthysis is a declarative testing tool that can keep up with your AYAC holding.

You state the properties you want your software to have, and Antithesis figures out how to check them for you. Verify your code as fast as agents can write it and ship with Groudon confidence. Head to insists.com/slash pragmatic. Kelsey, welcome to the podcast. It's so nice to see you in person.

They go get a shirt for you in the back. You know, what size do you wear? Uh men's large. And the one thing I liked about that job is you're dealing with real people that are in a hurry. I guess one bad part about the job, you know, a lot of people don't respect people who have that job. So they kinda look at you as just like this intermediary thing between them and what they want. But there's so many things that go into a restaurant like that's very efficient.

Um, you know, people have expectations and I learned how to run the whole store. So by the time I turned fifteen, I was a assistant manager. So nights and weekends, you know, other managers would leave. They would give this fifteen year old the keys.

And I knew how to do everything there, including close out the store, right? So you had to count all the money. You had to fax this huge spreadsheet to corporate every night. And then my mom would pick me up. And so it was really good learning how to really be responsible, even for adults at that age. So that's how I got my first dollar.

And so as someone who played sports, ran track, uh played football, played basketball, and is like, you know, there's this computer programming, not, you know, computer club technology student association. There was a class component and then there was after school component. And I was like, I don't know, man, this computer stuff, that's for the you know you know, I'm trying to be a cool kid. But the one thing I did, I really enjoyed it, right? So I I had a liking to AutoCAD.

I even competed at the state level in AutoCAD. So we we drove down and you compete. They give you a task and you sit in front of the computer. It was my first year doing it, but I really liked the idea of like taking a specification, designing it. And I probably would have gotten first place if I would have got the product to work because you also have to print it out so that the judges can review your work. So I got second place even though I didn't

And I was like, what's that? And it's like, look, we can, you know, every at that when at that time, you would create the SNC game, right? So it's basically get a magazine, copy and paste the code, and then you run it. And now you're playing Snake uh based on the code you wrote. And so you would toy around with this concept. So that was probably the first introduction to programming, was literally programming my my TI 86 calculator.

And so I decided to go to one that was near me. The first two weeks I was like, um, this is too slow. This is not this is not the pace that I want to move at. And also remember it's nineteen ninety nine when I'm graduating.

And so when you turn on the T V, people are standing in line for the next version of Windows. There's a lot of euphoria. AOL is starting to phase out and we're starting to touch on high speed internet. And and I was like, yo, look at look at how the pace this is moving. But also you're hearing the narrative.

Bill Gates drops out of college. These people are not necessarily glorifying the degree anymore. It's all about the skill. Now unfortunately for me, I didn't know anyone that was a programmer. I didn't know anyone that was like a system administrator because at that time all the systems like Sun Microsystems or IBM mainframe, those are still the things that are in the enterprise. So when I look at the job openings, I'm seeing a bunch of skills.

That I don't even know how to acquire. And so instead of going to college, you know, I'm still doing fast food, delivering pizzas at this time at Pizza Hut. And I remember going to a bookstore and they had the A plus certification guy. And I looked in some of the job posting and said, hey, you need to be A plus certified to take this support role or whatever it was. And I was like, you know what? That doesn't require college.

The book is only thirty five dollars. And I remember buying the book and It is an official certification process. It looks like it was part of the job market. And so I remember buying that book and reading it cover to cover over and over again. And you're learning all the fundamentals, right? You're learning about you know, how motherboards and how memory and all these things work and there's an OS component and then there's a little practice exam in the back.

And so for someone like me, having that fast feedback loop of like, you put the C D in, you take the exam, and even though it was multiple choice, you kind of felt like if I got anything wrong, I would just go back to the book.

and make sure that I understood what was written there and then you go take the test again. And it had a little randomization to it, so you couldn't just rely on absolute remembering everything. And I remember going to the facility to take the test and you know, you're in that little room And they want to make sure you don't cheat so there's a camera pointed at you. And you're just going through. And so the nice thing about those tests, there's no trick question.

Either you know it or you don't. And I think they maybe give you an hour, hour and a half. And I remember finishing that thing in like ten minutes. And when I walk down, you know, you wait, the diallop goes and they calculate your score and say, Hey, you passed. And then you walk out and you're like A plus certified. And that was like the first time in my career that I felt like, oh, so if you put the effort in, you can gain the certificate.

And when I got that certificate, I remember there was like a job fair where, hey, anyone that has A plus a network plus certification you can be part of the contractors that were replacing people's dial up with DSL at the time. Okay. And so that's how I I guess officially got into tech.

But when I looked at it, none of the people that at the time that I was looking up to, this is not the path that they seem to be taking. And so I had enough of school, right? If you're eighteen at that time, you're like, look, that that's enough of this. Because at the time I kinda felt school was this because it was so easy for me actually.

You know, it's like it was easy to get straight A's. I didn't feel like there was a serious challenge. So it's like, hey, I want to go and do four more years of this. And I would later learn that look, bachelors is a lot of the similar that you go through through K through twelve. you kinda remember stuff, you listen to the lessons. But then masters, you challenge the material. And of course if you make it to PhD, ideally you're adding something new to the field.

And I never saw anyone that has made it that far. So I never put that in part of my calculus. So but just having that immediate feedback loop of like getting this A plus certification And feeling like, oh, I'm ready to participate in the actual economy, the ecosystem. So to me I was like, this seems like a better path and it felt like a path that I would control.

Then when they made the shift to high speed internet, that means you had to actually touch the computer. And I think as a union, they're like, look, we don't touch the computer. We don't even go into the house. We we get to, you know, the D mark and we stop. And so they had contractors come in. And the contractors job were to come in, have to do a little bit of wiring. So if you had to run some cable, you did that. Create cat five cables, you did that.

Those were terrible. They always broke and you would always come out for a repair. Uh but for a new install, if you really wanted to do a good job, you install a a NIC, right? Uh cat five port on the back of their computer. And at that time, like we're talking Windows ninety eight. And so usually

I don't know, twenty percent of the time as you're installing the drivers, the computer would crash. And now you have a whole nother situation. You have to now troubleshoot getting this thing back online or back operational. But if everything went smoothly, they now had a network card and then you had an external modem that then you connected to, you know, the phone line and they had this high speed internet connection and then you connected the network cable.

And I did that for about, let's say, a year. And then I started doing the businesses. So you're going to people's homes, like you're going door to door. And then when you go to a business, you would hook up one computer, but there's obviously eight computers there, and only one of them has internet access.

And I remember at the time, you know, the business owner could be like a small insurance company. And they would say, like, hey, how do we get all the other ones online? And the first time someone asked me that, I'm like, I I don't know, man. I don't we put it on one computer.

And I was like, Yep, you could just write it out to Kelsey Hightower. And he was like, No, we don't we don't write checks to people. We write checks to companies. And I remember right there on the spot I'm like, oh man, I need a company name. And I just made one up, digital game. And they wrote that on the check. And I'm sitting there like, so how do you cash this? So I went to the bank. And they're like, Sir, you gonna have to get a business license?

The business license, you could just do business as you have to do this. I'm figuring out now I'm nineteen years old, like okay, I gotta go get a business license. I have to figure all this stuff out. So I do everything, I open a business account just so I can cash the check. But at that point I'm like, Oh, this pays more than this stuff.

And so I got really good at doing those network installs. I really got good at troubleshooting'cause sometimes someone gave them a USB modem, lightning comes, the USB modem is fried, and then you would swap them out for a network card. And eventually I decided I can probably do my own business. And I decided to get some office space. So I opened a small computer store right outside of Atlanta. I would buy parts from the distributors. I was just like nineteen, twenty years old.

And I wasn't buying enough to really qualify for an account. But luckily, one of the smaller computer stores I used to buy parts from gave a recommendation to the distributor. It's like, hey, this is our guy. He's just getting started. And they gave me an account and I was able to buy motherboards and GPUs and people would come over and like they'd bring their kid and they would have a parts list. I want a computer with all these things.

And we would assemble machines and you know sell them, but also it was the headquarters for all the other service calls. So I did that for like three, four years. You know, that ended up evolving into At the same time, or now we're talking like two thousand, two thousand one, a lot of the music studios were moving from analog gear and the large mixing board

to Pro Tools, right? The little rack mount unit. And they all need it maxed. They all need these conversions. So I added that to my uh abilities and then I had a small setup in the store and

artists and musicians would come in and say, Hey, we want exactly that in our studio And I would get the order and I added it to the list of things I could do.

It was a predominantly black audience in Atlanta. Okay, makes sense. And he did these jokes and they laughed. And it was one of these comedy clubs where if you're not funny, you have like three seconds and they will just boo you off the stage and that's the end of it. And he held us like wow, you survived that. That's um that's incredible. And you were pretty funny. And then we drove about an hour north and the audience is predominantly white.

And so on the drive there I'm like, there is no way you can do those jokes in this room. I gotta see how this is gonna go. And at the time they're only paying the comedians like fifty dollars. So you don't make a lot in the early days. And he totally pivoted the set and he held that audience too. And I was like, All right, I can be your manager. I know business. I know kind of logistics. I know how to, you know, make a plan together. And I did that for a number of years and

You know, he won some televised competitions. We went on the road with cup bands like Earth Wind and Fire and some large comedians from Kings of Comedy and Queens of Comedy. And I actually picked up some IT work with the company behind it called Latham Entertainment. They had been doing movies at this time. And so now I'm like, you know, doing IT for them and I got the small business, I got the comedian.

And so look, I was able to save a lot of money, but man, luckily I was twenty years old'cause I had all this energy, but I was working quite a bit. Eventually you settle down, you get a family. And you do the math, and it turns out people kind of over-glorify entrepreneurship. I think a lot of people believe. There is tremendous upside, right? The type of entrepreneurship we talk about with software companies, the the upside is crazy.

But when you're doing like selling parts or service business, unless you plan to open lots of stores and, you know, grow a larger employee base, it's not the same growth trajectory as software companies. And so I kinda did the math. After four years of doing that, I said, Look, I want to settle down and if you've been an entrepreneur before, you know employees get paid first. The owner gets paid last. And there were months where you get paid last.

Or you don't get paid at all. And now you're kind of drawing from savings because it's not their problem. But I did well overall. Like I did very successful. But I remember it's like, you know what, I think I'm ready. And so I looked around and Google had data centers nearby and I felt like I had a great combination of skills. I understood, you know, the racking stack part of the world. I understood the physical part of the networking stack.

I understood everything from Linux to Windows. I had an entrepreneur mindset. I I didn't think there was nothing I could not do. And I remember going to that interview and they were hiring like data center technicians. But what it paid. And in my mind I'm like, that you only have to work for eight hours?

You don't have to issue any invoices and you get paid every two weeks no matter what. No inventory? This is crazy. No way they're doing this. And so I go and I remember doing that interview and I didn't know Linux that well. And luckily for me.

I knew FreeBSD well. And as I'm answering the questions of like, look, I am not an expert on Linux, not the way Google was asking these questions. And it's like three people on the other side of this table just rapid firing. And I remember I was like, I know FreeBSD. I swear I got lucky. This one of the interview I think they had a

Free BSD tattoo on their leg, the little beastie logo. And I saw this logo, I'm like, oh, I'm saved. And so we started going down the Free BSD questions. And I pass and I get this drive in this data center.

And it was good because it's like, hey, I'm I'm working with my colleagues, but it felt a bit slow because you only get one job. You come in, you do this thing, and I got really good at it. Cause to me I kind of saw it as like a bit of a competition. Who is the best data center tech here? What are their metrics look like? How do I exceed their metrics? I want to learn how to do every particular thing in this data center because previously as a business owner, the more skills you have.

what money you can make. And then I just started switching jobs every three to six months because I just wanted to explore everything to just amass my abilities and doing the math. I think every jump was like a twenty five percent pay raise. I mean coming from a small base, it wasn't it didn't feel big at the time. But after a few jumps, my salary doubled.

Everything is immaculate. So a lot of people have worked in data centers and it's a mess. Wires are all over the place. You know, you're ad hoc adding and removing servers, but Google was systematic. Those machines came off a truck. They were wrapped perfectly. When you wheeled them into their spot, you connected the network cables. They would pixie boot. They would burn in. So part of the job was

No, you walked around with a crash cart, so depending on what your abilities were. Some people had pretty straightforward jobs. You had a crash cart, had all your tools on it and you would walk around and you would find machines that were needing of repair. So if you have 200,000 machines, it's okay if like 300 of them are broken. Right? The system can route around that.

And but if they were broken, they needed to be repaired. So you would go to Rack A server seven, you will pull it out and you will look at it and tell, Oh yeah, the SATA card is on fire. Like it's literally burned. It's burning. It was burnt. And so he's like, I c can diagnose this one with my eyes. It needs to replace the SATA card. But the thing is, You would go into the system and you would say This SATA controller needs to be replaced, so you will be making your prediction.

And then you would replace it and you would go through its burn in process. It would join back to the fleet. And then the way you were measured was how good were your predictions? Oh. Right. So you said it's a SATA controller and to me moving fast, you look at it, I think that SATA controller, I'm not gonna waste any more time on this one because I'm trying to get my numbers up. So I just swap the SATA controller, you bring on all the cables, you slide it back.

It goes through its process and you move on to the next machine.

were potentially bad. The technicians that were fast, they didn't waste time like running a program to do a extensive test of the memory. You learn how to use this little device and you would walk up to the motherboard. So the thing I would do is like before I do anything put this on the motherboard, get the reading, and I learn to trust it over time. Dem one, dim two are bad.

And the goal would be, all right, I have all this memory on my cart, you swatch those two dim slots, make sure that that's done first, reboot the machine, and you might be like, I think that's the only thing wrong. And again, you were put in the system. I believe I only need to do dim one and two. And then the way you were measured is how long before that machine gets kicked back to require. So if it doesn't get kicked back within, I don't know, thirty, sixty days, you did a good job.

If you didn't, your scores would be low. So for the technicians that were just reckless, right? They wouldn't even try and be you're just swapping the wrong part. You're swapping the wrong hard drive. And so your stuff is always coming back for repair. You were not efficient.

So I got to the point where I can maintain high nineties, but also repair, let's call it three times more machines than other people. So lots of machines, rate of return. And then I learned how to do the network switches and then there's power audits where you're lifting up tiles from the floor.

And you're making sure that everything looks good. You know, gotta be careful not to touch them because you could die. And so you learn everything about a data center, like the service loop. You're you know, you're running all this cat five cable and it has to have a perfect service loop. Fiber runs on a different part of the rack, right? So you don't ever mix these things. So as a person still like I'm in my early twenties, I'm thinking this is how all data centers look. It's not the case.

I started writing additional shell scripts to start and combining different functions together. It's like, you know what? I can't be moving this fast. There's a way for me to diagnose multiple things of the machine at one time. And so I would diagnose the SATA array. all the hard drives while I'm doing the memory component. And then when it rebooted, I will just run the script one more time as I'm putting my cart back together to catch that one more thing.

And once I started doing that, I can move as fast as I want it and the scores are right. So to me, when the scores actually match the things that you're doing, then it's a healthy feedback. And again, no one really talked about it unless you needed to talk about it.

And so I kind of leveraged it for a personal thing. I pulled it up in the morning. I kind of looked at my performance metrics. And in many ways, I calibrated my strategy based on this detailed feedback that I was getting. So I think I appreciated that level, that granularity back then because it felt like it was uh something that was helpful for me, not just my manager.

And so you would go along the server beach. So this is a spin out of Rackspace. So Rackspace is more like, you know, we'll buy a server and once you get it and it's a lot of manual steps. Rackspace was more of a let's automated everything. So the machine would pixie, some PHP scripts would run. If you ordered a RAID setup, then we would configure the RAID while the machine was net booted. And then we would put you on the V L land that you belonged on.

We would install the right operating system based on what you've ordered. And we just took a form, we just went through all of these steps. And then when we're done, and it took maybe about an hour, when we were done, you had an IP address, login credentials.

And if you wanted email and plus, you know, web size management, all these things, you got it. And when you were done, you give it back and then we put it back into the pool, ready for the next customer. And so when I saw how we were doing that, I was like, yo, this is Okay, you can automate things into end. And when the other thing that was important here We were doing things like updating the firmware for rate controllers.

Because once you pixie a server, now you're in memory and you have access to all the hardware. You haven't committed to an operating system yet, but you have enough to do whatever you want. So if you want to configure the Ray controller, and back then there wasn't like clean API. We were literally running curl scripts and command line utilities trying to get this machine into the right shape. And then when we were done, we would put it into the fleet.

So at that early age, I'm like, oh, there is nothing you cannot do. When we had to automate the Windows two thousand servers back then, we would just build tools that would literally screen scrape login to Active Directory, and then we would screen scrape mouse movement. So that way we can patch software on those Windows servers. So the concept of like I need a specific tool to do is like, no, no, back then it was like you do whatever's necessary.

Because these people are only paying like$99 a month. They don't have time for you to spend a whole week when they call random customer. You don't know their setup. You don't know their infrastructure. You have minutes to get them back online. So everybody learn to move quickly. No complaining. When you get that ticket, it's on you to figure it out. Maybe you link to your teammates for help.

So I kind of learned how to move fast. But the the inflection point came from I started that job in tech support. So people would call. My MySQL isn't working, DNS isn't working. They can even describe what isn't working sometimes. And so I realized that we were all in the phone queue. When the phone would ring would it would just round robin between everyone. Then if you couldn't solve it fast enough or you couldn't solve it at all, you created a ticket.

And when the tickets sat there and then ideally you get to it later, but the ticket queue would just get long. And when a shift change happened, we just had all these tickets piling up. And of course customers are now mad. Three days, no response. So one day I said, look. I'm just n not gonna log on the queue. I'm just gonna resolve the ticket.

And I'm back in that entrepreneurial mindset. I'm building little scripts to take a ticket. I see the issue. This is my SQL issue. We need to vacuum the database. This is easy to run that one. Ticket close. Hey, sir, everything is good to go. Please try again. Close. Oh, this is not even an issue. Close. Plus, oh, we can upgrade PHP. Close. And then the ticket queue is zero. It's just empty all day.

And so someone's like, hey, Kelsey's not logged into the phone queue. He's not even on the phone. He's not taking calls. My manager pulls me in the office like, Hey Kelsey, why are you not on the phone queue? I said because we don't all need to be on the phone queue. We can just have one person making the ticket queue stay at zero.

And then I would tell my colleagues, hey, hey, look, if you can't figure it out fast, just open a ticket super quickly. I will take care of it. So some people specialize in Windows. They got a Linux call, they didn't know what to do. I said, Don't worry about it, just put it in the queue quickly, move on to the next call. I got you. And so I became super efficient. So I explained this process to the manager and he thought about it. His name is Mike. And Mike was like

Yeah, I like that. We're gonna change it. We're gonna have a couple of people stay out of the queue. But the promise is that cue has to stay zero. And it's the first time I learned the difference between activities and impact. activity is you being an all star answering a bunch of calls, but the ticket queue for the team is still high. You making this jump and saying, look, maybe I stay out of the queue

And my promise to my colleagues is impact. The ticket queue is zero. So on Wednesday when we do the team turnover, we're handing off an empty queue. Now of course I didn't teach them that because I didn't think about it. And when we would come in, the queue would be high again. It's like, yo, whoa, this is not you guys keep handing off a bunch of

Burden to our team. And of course we would clean it up. But then the management team was like, yo, everybody's going to do it. And then I was like, man, you can change the process. So that was like the huge inflection point. I think the next one was more about being a mature person. Stop job hopping so much. Because again, after a maybe a few promotions and some impact, I'm now off to the next thing.

And I didn't have to necessarily live a long time with some of those decisions or, you know, stay long enough to really impact more of the culture. So when I got to financial services, it was the first time I got restrictions put on me. Right? Because in these companies, everything is about moving fast. Google, we gotta move fast. We have to compete with the big guys. We're gonna do with cheap hardware, smart people.

any means necessary. Web hosting, we're not charging a lot, margins are thin. You gotta move quick. Financial service like, no, we're making money over here.

And uh I remember seeing jobs that I used to be afraid of five years ago. The same job that made me want to just go get a A plus certification and open a computer store instead of even trying. And so I looked at those job descriptions again.

And it was like you need Linux the light, I got that. Even at that job I got Red Hat certified, right? So I was like, I'm I got all the qualifications now. And I remember the job, I didn't know how much it paid before I went, but you had to wear a shirt and tie. It was the first time in my whole life I had to wear a shirt and tie.

Yeah. And I remember my homework teacher, she hemmed my pants for me. Right. I was like, Hey, I got a job interview. Uh does this look right? I'm driving up on my lunch break and I get there and you know, this is enterprise. I'm like, wow, this is This is the big league. I didn't even think Google was big league. I thought financial services big league. This obviously if you have to wear a tie to do your job, you must be doing something so serious.

That you need a shirt and tie to do it. And so I get to the interview and I'm sweating like what they're gonna ask me stuff that. I've probably never heard of or seen before. And they started asking Lennox questions. I'm like No, you gotta use this flag, that's not the right flag. No, look it up, that is not the right flag. You can't do that with grip. No, you gotta pipe it this way. The PS table doesn't show you that. They're on that version of Linux, not with that kernel.

And I'm like, It's too easy. And part of me is like either I'm really good or something else. And so I'm thinking like, yo, that was maybe there's another round. And so I'm driving home And I'm loosening up the tie and I'm calling my wife like, Hey, I I think I did a really great job. Like, you know, th I think and And this has been a pattern, you know, I I get good at something and I find a better job. I got good at something, I find a better job. But now it's like, this is like a career.

And so I'm talking to my wife and I was like, Hey, I gotta call you back. And it's the recruiter calling. Hey, you have a second? It's a hundred percent. Uh they want to make you an offer. I'm like, you know, make you an offer is different. Like it's usually this is how much the job pays.

And so I got to I got to that company Total Systems Thys. And um it did feel a bit slow. They had run books, change windows. Everything was regulated. This is a financial institution. We're processing credit card payments. We're doing work for the government. We're doing all of these things. And I remember the team was just everybody just moving at this pace.

And then it's a bit disappointing. And I think did you not not feel a little bit like, am I missing something? Like this should be, you know, higher. This should be, you know, the interview was okay, maybe not as hard, but it it it was something.

So I didn't know that in the beginning. So I'm just like, oh, everybody's moving a little too slow. I was seeing how they were doing deployments. I'm watching how they're provisioning servers. I'm like, you know, you can automate this whole thing because I've done it multiple times. And then I learned to be a little bit patient. It's like, all right, hold on.

I learned how to deal with the no's. I learned how to deal with the executives. I learned how to talk not just to engineers, but to the senior leaders and get their trust. And I won't tick about everything that I did there, but I was there for about three years. So the job had me stop. And I remember there was a task. where we were using Apache and some Java plugin to talk to our J Boss instances. And so they were using a lot of memory per connection on the load balancer.

And so we got a new we're moving off the mainframe. We were moving into this new, you know, Java world and the servers just kept falling over. We couldn't handle all the transactions. And I'm just watching the team go into these change windows and then we fail. We would fail. We would fail. And the CTO at the time was like, you know, this is costing us money. These we're getting chargebacks. We're having to pay out penalties.

And I was like, I had a dev environment where I was using Nginx. I got rid of all this Java stuff. Like it's it's just HTTP. You don't need a Java connector thing. I'm reading the spec. You don't need it. And my memory usage is a fraction.

This thing will work. It's like, I don't know, Kelsey. This is not certified stuff. It's like, no, no, no. I got it from Red Hat. RPM install engine X. It's in it's in Red Hat. It's like okay, it's in Red Hat. Okay, sure. I said just give me one chance. Yeah. And about after a week, they gave me a shot. And they let me be in the change window. And

There is no hiding behind the team. It's almost like everyone's sitting back like, You got it. Now luckily the leadership was like, We are supporting you and we're putting our careers on the line as well. I mean, they probably would have been safe.

But after about two hours, I got everything working and we just watched the memory pressure just drop. So if we were at ninety percent on utilize, I don't know, let's call it thirty two gigs and we're just blowing the stack every time everything's crashing, this thing is hovering at like

Two gigs of RAM. And everyone's like, yo, like, are we getting peak load? Is this legit? And we're all just sitting there like not sure. And the way we used to test the platform is we would someone would drive to the gas station, get a credit card, buy some gas. We want to see the transaction land in the Oracle database.

Let's see. So ten o'clock goes by and we look around and there's no Nagios alerts going off. I'm like, man, we might be we might be good. You know, me, I got my I got top going. I'm looking like memory usage is holding steady.

And'cause he probably is like, Kelsey, if you get this to work, I'll buy you a steak dinner. And I at the time I still am. I was vegetarian, so I don't eat steak, but here's the here's what I would like instead. You gotta buy pizza for the whole floor. Now I don't know how strategic I was thinking, but I figured that I could score some points with the whole team from turning it into a I succeeded to a wee succeeded.

And then people eating together is one good way of doing it. And I remember after it worked, he was like, What's the order? And I just ordered, you know, pizza from one place and we just put it in the middle. Everyone's like, what's the pizza for? Right? Because usually you only get this for special events. Like for the thing we did.

And the next day some more pizza and some more pizza. And so when I got that accomplishment, I was like, okay, this is what maturity looks like. It's not about just having the right solution. You literally have to get consensus, buy in

make sure and then your reputation's on the line. And I got that change in and just kinda changed the way I thought about what success looked like. And so that was the inflection point. And then of course I brought Puppet into that organization, repeated a similar process of automating everything with this tool.

Shopify started running on BuildKit in 2015. Airbnb, Canva, Uber, and OpenAI have all been using Buildkite for 7 to 12 years. Throughout the 2010s, the hardest CI problems in the industry for global software leaders were getting solved on BuildKite Pipe.

That's exactly what makes BuildKite the CI platform most in tune with what's happening right now. Gen AI Throughput is pushing everyone's build queue to its limits. Coding agents are pushing commits 5, 10, 50 times the volume that your defaults were built for. The same architecture that absorbed Shopify's traffic at scale five to ten years ago now absorbed a shared push from Entrophic, Cursor, Meta, Mistral, Cohere, VLM, XAI, Lambda, and more, who are all built customers.

Where other CIs are cracking under the weight, BuildCat is running about 1.4 billion job minutes a week. Agents run on your infrastructure or on BuildCat. Every artifact and log is captured. And when something fails, you see the why immediately. If you're hitting ceilings on your existing CI or downtime is becoming more problematic for your delivery, head to buildkite.com slash pragmatic.

30A all access trial, no credit card, and an actual human NGR on the standby. His name is Ola, and he is very helpful. I'd also like to mention our season sponsor, Century. AI agents don't have a feedback loop from production observability logs, but should they have it? I mean, we probably don't want to build a system where after each production error, AI agents automatically push out a fix into production without super.

SentryMCP is one way to plug in Sentry to agents that support the model context protocol. Clot code, cursor, codecs, VS Coded Copilot, they all do. After you hook up the MCP server, you can do some very useful things. For example, you could do this. When an already resolved Sentry issue resurfaces,

You can kick off a cursor agent to investigate their regression, read their relevant code, and open a PR with a suggested fix. There's a little work involved to get all of this going. You need to connect Sentry to your code repository, add SentryMCP to cursor.

Define the instructions for Cursor's agent to investigate, configure the trigger that launches the automation, and test that it all works. But once you have it up and running, you can get regressions fixed faster whilst they're reviewing every and all fixes.

This feels like a sensible and helpful use of AI, NCP, and Sentry to me. Check out sentry at sentry.io slash pragmatic and start monitoring and fixing regressions today. And with this, let's get back to Kelsey and when he introduced puppet into his organization and automated DevOps processing.

And you did that every day over and over again. And that's where the saying that I I say sometimes, some people have twenty years of one year experience. Some people on IT have been doing that. Twenty years on the row. They never made the leap to automation or learning new tools or skills. And so I saw like, man, maybe I can bring in a tool. And I brought the tool called Puppet at the time, zero two, four eight.

And this is right before DevOps becomes a word. But it's like, all right, I'm starting to learn how to write code like the for the Java production stuff. I'm writing Puppet, you know, manifest using the Puppet DSL. Sometimes you have to write a little bit of Ruby to build a new resource type.

And I'm contributing and also I get introduced to open source, like, hey, there's something there that doesn't work. I'm gonna contribute it upstream. So I'm doing all this behind the scenes, getting ready for act two. And I remember my manager went to a conference and he came back and was like, Hey, Kelsey, I finally know what you're doing. I was like, What's that? He was like, uh you're doing DevOps. And I part of me was like upset. How do you get to name what I'm doing?

And so, you know, everyone's talking about DevOps and he's like I met a guy named James Turnbull. And James is an Australian dude. He worked at Puppet Labs and he wrote the first puppet book that I bought and read. And so I'm like, James Turnpaw, that name sounds familiar. Oh, I'm looking at my desk. James Turnball and Jeff McCune, so he's one of the co authors of this book. And it turns out James wrote a lot of tech books back then.

And he's coming to the office and my manager wanted him to check out how we were using Puppet. And I was like, Okay, so you know, this is a world class expert. And by that time, I had Puppet hidden behind Jira tickets. And so you could just open a JIRA ticket. I had RPMs for everything. You picked the RPM you wanted from a drop down. You picked your environment from a drop down and magic happened.

And you got results. Most people didn't even know Puppet existed. Most people never touched it. They just knew they can get anything on demand. You could have.

And we will take the output and update the ticket. And so PMs, developers, they just got what they want damn near instantaneously. So we built all the systems. They didn't know nights and weekends, I'm contributing to Puppet. 'Cause you know, wasn't allowed at work at the time. This is two thousand seven, two thousand eight.

And James turns to me and he's like, Kelsey Hightower? Oh, we know you. We love your contributions. And my manager's just like, what contributions? We're we're not doing contributions. And James is like shaking my hand and he should just talk it to me like we're friends because the thing is I have been working with the Puppet Labs team. Through like open source contributing to Puppet. And so we get up there and you know my manager's like, hey, let's show him our setup.

I was like, all right, James, we have a lot of Puppet manifests over here. I'm using external node classifiers. I'm setting up Brighton config from data. And I read M Mark Burris' promise theory and got it all dialed in, but from an interface perspective, I just didn't want everyone to have to learn Puppet.

So we hooked it up to Jira. I'm showing him this entire workflow. And he's just sitting there like Yeah, I have no recommendations at all. Like this is this is amazing. And here's the thing that kind of the game changer. So he came all the way to Atlanta. We're forty five minutes from the airport. He spent time with our team. And uh it's the end of the day. And then he's on the way out.

And he's about to call a cab to go to the airport. And I'm like, no, I c you're in my hometown, I cannot let you go to the airport. I will give you a ride to the airport. So I get in my car and he's in the front seat. My collie's in the back seat. He takes off his tie and rolls up his sleeves. I can see all the tattoos. I'm like, oh.

So you're like a legit technique's like, yeah, I'll do all of this. I just did this for you all. This is your dress code. I was like, oh James is cool. He was like, man, yeah, you guys are doing great work and you know now I'm talking to the real James Turnball. And uh he invited me to give a talk at Puppet Cop. And I went to go give that talk. It was my first like real conference talk.

And I talked about m some low level kind of puppet integration stuff and I had a little comedy in there. It's like the first time I felt like I could just be myself on stage. Before I came home, Luke Canise, the founder of Puppet, We sat down at a little coffee shop in Portland, Oregon, and he was like, you know, would you like to work here? And I was like Yes.

Of course, hundred percent. And of course there was a nice salary increase and I could work remote. I'd have to leave Atlanta immediately. And I remember coming back and uh my manager really appreciated all the progress I was making over the years. I kinda made a name for myself. And I remember he gave me a nice little raise just out of the blue. And he slid this envelope to me and I opened it and it was a nice number in there. I was like, Oh, that's great.

I was like, but here's a thing. I'm resigning. I'm going to go work for Puppet. And then he said something that was dope because we had a kinda, you know, difficult relationship depending on how it went, but overall he helped me mature. That's one thing I will always say. He helped me really be mature. And uh he was like, I'm surprised you stayed here this long.

Right, as like this ultimate compliment that you stayed here beyond the just getting better or making an impact. You literally changed this culture. And the full circle moment happened like seven years later when Kubernetes came out, I remember coming back to that office. And they had already been running et C D in production, Kubernetes in production, and even some of the old tooling I had was still running.

And to me that was like a really important feedback that sometimes if you make an impact lasting impact is way different that even when I wasn't there, they still had the culture to say there's new technology and we know how to bring it into the stack.

And really at that time, we're talking 2011, 2012, in my mind the competition was only between Puppet, Chef and Ansible. That was it. And so I'm all in. And I'm working at Puppet Lab. And so we're kinda in some ways I felt like we were dictating the future.

We were going to companies and getting them to understand configuration management. We were talking about the benefits of like speeding everything up and, you know, doing all these compliance. We were moving people from SharePoint to executable code to implement these things. So we thought We were just getting started. There was no world where we then think that we needed ten more years of that.

Right, we even got money from being where to integrate Puppet into that. Chef was being integrated into AWS at that time. So this was like the thing that finally we're finally getting there. We just got to teach. system administrators become engineers via DevOps and start embracing these tools. And then the first thing that I saw come out was Golang.

Mm-hmm. And I remember sitting at my desk and I was like, you know, at that time we were hitting performance issues with Ruby. Yeah. Right? The global lock interpreter, you can't do more than one thing at a time that easily. And so we m moved to things like J Ruby. We even did closure for some part of the puppet stack. And I remember we were thinking about should we start doing stuff in C plus plus?

Should we start doing stuff in C? But the problem is cun contributors. What would we do with all the Ruby contributors? Right? And I remember downloading Go for the first time. And the thing that sold me on it. I remember um one of the prototypes that I built was Factor. So Factor

is one of the agents in Puppet that gives you facts about a machine. So this is Red Hat, this kernel version, this is what the users look like, and then would give that to Puppet so that your configuration code had context on what to do. And that was written in Ruby and sometimes it will run slow because you're reading all these files seriously. And so I remember I was like, all right, let's try this goalingout. And I remember I wrote the code on my Mac.

compiled it, SCP'd it to a Linux box. I was like, oh, this is crazy. And I remember running all the facts in parallel. And it came back in a fraction of the time. I was like, yo. We gotta use Golang for stuff. And I remember the team was like, nah, it doesn't work for Solaris or AIX because Puppet was now moving into the enterprise. I was like, that's the criteria? No way. But I got it. Right. I've gone through this maturity thing before. And then Terraform comes out. So before Docker.

Terraform is trying to is starting to challenge things a little bit, right? And it's starting to use things like, go, at least that was on my radar. So Terraform is like, who cares about the node? It's all about the API. And all of us come from the server world and everything's about an agent on a node versus the cloud starting to take over at this point.

Look at that, they're using Go for this. And I remember when Docker came out, Puppet at that time was the one pushing innovation. We're asking people to think a little different, get out of their comfort zone. And then Docker comes out and most of the office was dismissing this as like, Nope, this is a fad.

They don't even have config management in this thing. They don't really understand enterprise. This is just kinda not like a toy. They were not being disrespectful. But no one saw this as a challenger to what we were doing. And I looked at it and I didn't get it immediately. Until I left. When I left Puppet I built this tool called Comp D. I went to go be a VP of engineering and I got to write Go code. So we started roughly so we looked at all the Java heavy usage.

And I earned the trust of the team. We started rewriting some of the microservices in Go, shrinking our cloud footprint. And we sunset it and got it all into production, the Go code. And I was like, we don't really need Puppet anymore. And I open sourced this project called ConfD and it would pull variables from etcd and generate just enough config, just the parts of Puppet that I thought made sense. And then Docker was out.

And then I was like, wow, we can probably stop moving Python files around. We can probably package them up, not in our P.

Even today, don't package their third party apps like the apps that a development team would write. You're like, no, we just put CI C D, we'll copy them over there, maybe put them in a tarball. But we usually never went to making official Debian packages or RBM packages. That puppet meant you didn't have to go through all of that work and you can still end up with a packet.

something that was repeatable. So all the stuff we used to do with Python and Virtual Int, all the things we used to do with Ruby gems and you know the virtual environments we have for that. We got rid of it. You squish it all into a Docker container and you got rid of a lot of dev tooling. So I this is why I think it resonated so much with developers, because we cleaned up the mess of working on multiple projects. And so

I was attracted to that. Comp D was compatible with that. And then CoreOS was the thing that I was like, you know what? I think I know what I wanna do. But I didn't understand distributed systems, not to the degree what Coral West was doing. So when I went to Coral S. So Core OS was like Google's infrastructure for everyone else. And so at that time we did there was a tool called Mesos.

Um the board paper had already been published. I try to read it when I was at Puppet Laza. I don't understand any of this stuff. Mace was hard to install. I couldn't justify it. But we would see kind of the rumblings of the Twitter folks talking about the distributed system and all this maybe big data stuff is going around. But I was like, I just don't understand it. It seems incompatible with the stuff that I think about. Even as someone who's worked at Google,

that had a system like this, right? They had Google cluster file system at this time. It didn't seem as complicated as this Mesos thing. So I was like, ah I dismissed it. But what Core West did was build on top of Docker. Coroas is like, what if we had an operating system that only had Docker on it? Everything is written in Go. We can have a little key value store where you can put your config and we will just synchronize it to all the machines. And that was so compatible with the comp D way.

And so I'm looking at this Coral West thing, I'm like, yo, this looks a little bit more like the future. Because that's what we used to do. You know, I had at this time I was a s software developer at Puppet. I was a software developer, VP of Engineering at the other company. And I was like, you know what? Ops can learn a lot from the p uh darker way of thinking about the world.

Because as a system administrator, we always try to make the OS small, remove things you don't need so it can be secure and repeatable. Docker was like, for the things that need to change, just put it here and isolate it. And Coros is like, what if you had an operating system designed only to do that? And I met the Coros team at GopherCon, uh a a Go language conference. And they saw me present. And what I learned from the puppet date.

Every presentation's an interview. I don't think a lot of people think about it that way'cause people are looking at you on stage and They get to see what you're about. And so at GopherCon, I remember also Rob Pike and the original creators of Golangar and Oh, this is like the first GopherCon. Right. And so the original creator, Russ Cox, all these people I look up to, Brad Fitzpatrick, And they're all in the audience. And I remember I had a talk.

Two things. I'm sitting in the audience waiting for my turn. I'm number four on the list or something like this. And the two people who started GopherCon, they're just from the community, Brian Kettleston and Eric Saint Martin.

And they're new to this conference scene, like you could tell, right? You know, they weren't the best MCs in the world. But they were like, Hey, welcome to Gopher Con and at the point in time, Ruby is still dominating. And down the street there was the Ruby conference going on. And we were just in this other building. And I remember the first talk, they introduced Rob Pike and he gave this amazing keynote around Hello World.

He went up all the way down to the compiler, he went all the way up, and he described how the language took shape. It felt simple on the surface, but it went super deep. And it's like Rob Pike had also come full circle from his ATT days.

through Unicode, through all the stuff that he's ever done. And now we get to see one of his best works, Golang. And my talk was titled Golang for System Administrators. I wanted them to see that there's a better way. The way they introduced Rob Pike though, I was like, oh man, this is Rob Pike, you gotta You gotta have more energy than that for Rob Pike. So I'm sitting in my chair and I don't know them, but I'm sitting

With a buddy of mine from that company where we rewrote everything in Golang, Billy Cleek, we're sitting next to him and say, Hey Bill, I'm gonna go ask them, can I be the MC? He's like, What? Okay, so I'm winning. And I walked to the backstage it's like, hey Eric, uh, Brian, uh, can I try my hand at being the MC? And they're like, sure, right? And they knew who at least who I was because they accepted the talk.

And so I don't know if it was after Rob, but I came out next. So like, hey, I'm Kelsey Hightower, you may not know me, but I'm gonna be your MC for the rest of the day. And I don't know what happened because it was my first time doing that. I'm cracking jokes. I'm having fun. And I say, hey, from here on out, when anyone comes to the stage, it needs to be loud in here to the point where we can get kicked out.

And so I was all right, so our next speaker, and I think it might have been Derek Colston, you know, he's like a Googly, he was talking about some ghost stuff and I remember it got really loud. And then they come on stage. So if you've ever a speaker ever before and it's like you walk up to basically a standing innovation.

You're energized, everybody is excited. I don't care what you do, the energy level is high. And so he comes out, he's having a good time, the audience is having a good time, and then eventually it was my turn. And I realized like, who introduces me? So I go onto stage and I was like So we're gonna try it like this. I'm gonna go back and I'm gonna come out and you guys can make a lot of noise from my talk.

And then I remember just like sliding slowly behind the curtain. So everyone's now laughing. And then the music comes on and I walk out surprised, like, hey, and everyone's clapping. And I do this talk. And at that time it was live did more or nothing. And at the time they had this thing called Go Present. So Rob Pike team, they wanted to make sure they could use Go for everything, even generating a slide deck presentation.

And I remember in VMware Fusion that runs on your laptop. So if you want virtual machines on your laptop, you can use desktop parallels or VMware Fusion. The thing about VMware Fusion, on your Mac, you can create virtual machines, but you can also swap out the network card. And I did one where you can have a network card that talks to iPixie. So I wanted to show them how it um boot up multiple machines from a GoPixie server running on my laptop.

So part of that demo I was start the I had this kind of snippet of my Pixie server and I hit the run button. So now my Pixie server is running in the background. It's like so what can you do with it? And I remember looking in the front row and Rob Pike and team were just intimately looking at this thing, like, wow, this He just started a Pixie server from his laptop. From GoPresent the slide deck tool.

And so this thing is running and I'm like, all right, let's bring up VMware. I gotta make sure we switch the adapter to the one that supports iPixie, gotta do some firmware thing and then I remember I booted the VM. You can see the logs in the GoPresent of like HTTP handing off the image, giving in the IP, and the virtual machine is booting up. And you can just see the amazement on the audience face like

Did you just do that? And then I booted another thing in the thing and I'm going through why I think that this is a game changer for the craft that we have. We finally have a tool where we write high performing things. with the simplicity of imperative programming and we can go beyond just scripts. We can actually build systems.

The audience was dialed in and it got loud like after you know things were working, people were clapping. I look in the back and there's a whole bunch of people now standing. And I recognize some of the names because they're from the Ruby community. These are people that are writing the Ruby book.

And then, you know, after I'm done, there's a break. And I walk back there. They're like, hey, we saw on Twitter that you guys are just like going crazy over here. It's out of control is electrifying. We left the Ruby event. to come here. And so I felt like, man, it had arrived, but guess who else was there? The CoreOS team. The Core West team was watching me because I was pixie booting CoreOS.

This is what I mean is every job is an interview. The team was like, all right, we can see that yeah, you should join the Core OS team. So that's how I ended up at CoreOS. This is how I really felt like the container movement had legs. And this is what maybe a year or two before Kubernetes comes out.

it's hard to value skills you don't have. So a lot of software engineers are terrible. If you put a software something, they can't talk. They can't simplify concepts. Maybe they can write code really well, but this is a set of skills that they may or may not have.

And so when you see someone like that you're you're you're questioning them. And I remember giving a talk one time at Strangeloop about et C D and Core S and someone was like Do you understand uh is is etcd like a C A system or an A P system, like the Capther?

And the question was kind of loaded, like, we don't think you even know what that means. I was like, um, etc. is going to always favor consistency. He was like, That's not correct. The wrap paper, blah, blah, blah. I said, Listen, you said et C D and let me show you.

This is the raff log. This is my three nodes. I'm going to turn off two. And you notice I can't write any keys. So availability has been sacrificed. Consistency is being preserved. I'm going to start another node. There's going to be a handshake. There's going to be quorum. I will be able to write keys. Now it works. That's the cap theorem. In reality.

So it doesn't matter what the RAF paper says. Are you talking about a RAF log on a single implementation? RAF doesn't talk about cluster membership, leader election, how it's implemented, and what you should do in the different modes. That choice is yours and this is how etcdia is implemented. And I remember he was like, Oh shit, this guy actually know What he's talking about?

Being at CoreOS, we were working on our own fleet management system called Fleet. We were using System D and we were trying to synchronize configs through et C D and remember in a CoreOS cluster. All the nodes communicate via etcd. So imagine using systemd for those that have never used systemd, you put a unit file, like I want this process to start with these flags, you know, bind to this port. And you put it in a directory and then system D will start it.

And if you had a thousand nodes, of course you could SCP that file to all one thousand nodes, and then there you have it. So we decide instead of you copying all the files to all the nodes. just put the unit file in etcd and the node that should run those things would then pull from etcd and just run those unit files. And we called it fleet. So we had our own vision of giving people a distributed spouse or distributed system.

About a year goes by, Kubernetes comes out and everyone was like what And we're all we got like a day notice. So the Google team reached out and was like, hey, tomorrow, we're announcing this thing. Here's the GitHub repository. Here's you guys are under embargo. Don't talk about it. And so I'm thinking, we're not part of this story. Like our we're our names are not in here. It's just as Google and Red Hat and they're gonna announce it at DockerCon. There's no Core West in here.

So what can we do? And I remember I stayed up all night. I got access to the GitHub repository. I'm reading all the go code, trying to figure out what all these binaries mean because there's no docs. Yeah. And so I got everything working on CoreOS. So when they did the official Google announcement, of course there's a famous Docker keynote where they unveiled it to the surprise of even Docker to some degree. And I remember they post their number one on Hacker News, and then we post.

Hey, what they just said? But here's how you run it on CoreOS and some examples and commands. And I have to do a few patches to get it to actually work and I have to build some binaries to get things glued together.

We got CoreOS, we're on bare metal, so I have to pixie boot some things, but I think if you put all these things in the right place, there's this etcd thing that's ours. We know that, so we know how to use that. But I think the API server connects to that. There's no volumes, there's no config map.

So all you can do is get this thing stood up, and then you can submit a config, and then it will just basically use Docker in the background. Okay, I can document that. So we get everything to work and I write this nice little guide. Someone on our team, they publish it to the official Core West website.

So then we launch that on Hacker News. And then we go to number one. And everyone is like, Google just announced this thing. We don't know what it is. And then Kelsey launches this thing. We now know what it is. We know how to install it, we know how to run it, so now people are downloading Core OS just so they can play with Kubernetes. And I had a keynote probably in a week or something.

I don't know what I submitted to the conference because usually you submit like months ahead of time. I was like, hey, I know this talk was supposed to be about this, but it's not today. It's gonna be about Kubernetes. And people are like, what's that? It's like, yeah, it just got announced. I'm gonna show you. And I started giving people live demos.

Of Kubernetes and how to make it work, you know, using Core OS and all these things, but the team still wasn't sold because we had our own roadmap. Yeah. And also at the time

Yeah. And so we didn't know like maybe no one's gonna care about this either. It's only when I started going home and starting doing small contributions and starting to read everything and getting a feel for it. I was like, No, there's something here. And so for maybe two or three months, and luckily the founders of chorus were really nice. Alex Povey and Brandon.

They were just like they didn't get mad or anything. I was contributing nights and weekends, kinda like I was at that other company. Uh all of my keynotes were more like Kubernetes plus Core OS. And I remember at some point it was inevitable. We all got in a room. I became the product manager of CoreOS at the time and Alex is like, I think you got the vision here and we got everybody in the basement in San Francisco on the office we were and we were like, Hey guys, all in on Kubernetes. Fleet.

Deprecate it. All these things that we're building, deprecate it. We're gonna go all in. And I'm glad we did because um Alex Povey came up with the name operators, which is like a core idea in the Kubernetes community. We put all this effort in there. Me and another guy worked on a thing called C and I, which is the networking layer for Kubernetes. And what Kubernetes really meant for me was that previous fifteen years of experience as a practitioner in the data center, learning promise theory,

Learning Puppet, where it works well, where it doesn't, and then understanding that Puppet wasn't the only way. And then making going through all of these loops. And so when I ended at Kubernetes, it felt like This would be the thing I would build if I only knew how. And that's the way I explained it to the rest of the world. And at that gopher con, maybe the next one There were people from the now early Kubernetes community. It's a small company called Kismatic. There were kind of a

Core West competitors to some degree, but they came up with the idea like we should have a coverage just like GopherCon. We called it KubeCon. Joseph got the logo going and the Kismatic team kind of put up the money for the first event. And we really welcome the entire community. And now it's been twelve years later and the CNCF has done a fabulous job of keeping it going. Now there's like what 13,000 people here in Amsterdam keeping that thing going.

Maybe the Achilles heel to Docker Swarm was its design. They tried to take the Docker API, which worked really well for one node, and expand it across multiple systems. And it was not the right API to scale to another type of thing that we needed. And so they kept trying. They tried to add storage. They tried to add networking. But the Docker API was never meant for that.

And so the Kubernetes team was smart. Instead of trying to say Google's better than everyone on everything, they did a couple of things correct. Let's just use etcd. Let's just use Docker. So you take those two things and you take the experience of the people who wrote the Omega paper, which is kind of thinking about what would come after Borg and at least the things like Mesa.

It doesn't need to be that complex, especially for certain workloads. You don't need this like high performance, over-engineered thing. There's a simpler way to do scheduling, especially if you can give the scheduler a bit more metadata about the workload. They're also bit game changers now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers.

And so I think that's the number one success criteria that we were already off to a running start because you could just reuse the same Docker containers. You didn't have to rebuild a new image thing. So given that What they tried to do in the early beginning was just fill in the gaps. And in many ways, yes, it's a new system, but it fills in the gaps. The one gap that they filled in was Docker had an entry point. So if you needed a Ruby app that needed Nginx and your process,

You used to have to write a little shell script, the entry point script that would do all this magic, almost imitating an init system. Kubernetes is like, no, no, no, you don't need to do that. You can just make separate containers and then Kubernetes will run them as a process tree. And so for many people it's like finally now we can have a clear way of thinking about application architecture.

you have to specify exactly the containers you want, how much memory that they need, and then we have the status field to tell you if they were running or not. And you would take this data object that you could write by hand give it to an API and then the control loops would operate on this state. So that means it didn't matter if you had Ruby, Python, or anything. You can just take your IDE, write some YAML,

give it to another tool, manipulate the YAML, and then pass it down to the API server. So you can build any combination that you want it without having to be a compiler first. That to me was a fundamental game changer that I don't know if a lot of people understood why it felt very easy to onboard the Kubernetes. Kubectl apply object.

Marathon, but you had all these other tools sitting on top of a thing. So an extension in Mesos was heavy, almost like a whole nother system. The thing that makes Kubernetes powerful, there's a data model. We gave infrastructure a type system. So instead of imperative shell scripts, you finally had type.

So if anyone's ever come from like Python to a typed language, types do a lot for you is in terms of cognitive overhead. Like you really know what goes into this function. If you put the wrong thing into this function, it doesn't even work. Like you c you can't pass a string where an integer needs to be. Kubernetes brings the same semantics to infrastructure. And finally, now it's much safer to automate things because you're gluing together things that actually have structure and type.

So instead of trying to evolve Kubernetes to do everything, Brendan Burns, I remember sitting next to him. He's like, Kelsey, let me show you this thing. you can extend Kubernetes just by giving it a

description of what you would like your object to be. So if you were thinking about this from the rest world, hey, I need a user. Here's the crud operations and just give it to the thing and it does everything else for you. And so when that came out, it's like so if I want to manage Like yeah, you can describe a firewall and you give that to Kubernetes and all the tooling work.

You can now say coop CTL apply a firewall. And so now we got tools like cert management where if you want a certificate from Let's Encrypt, you can just say what domain you want, where it should live, give it to it, and now you had a first class extension. You didn't have to uninstall it. You have to make some magical binary and it really didn't matter what what language you wanted. Because once you put the data model in place, you got the machinery.

And if you care if you like Python, you can have Python running a loop, grab the data, and then make it so. If bash was your thing, you can literally pull the config using a bash script and make it so and just update the status field. That opened up the entire ecosystem. So Cisco could come in, do what they wanted to do. Red Hat can come in, open shift, and do what they wanted to do. To me, that was the game changer that brought the rest of the community in.

And then once we get in, we're thinking about career progression. Dev, senior engineer, principal engineer, distinguished engineer. And we spend almost our entire lives on that trajectory. And our field is so young That some of our pioneers are finally like no longer here for the first time. We're not used to that.

And we're not really used to people retiring. Like Rob Pike just retired. The concept of an exit for an individual contributor or a leader in the tech space is new. We don't we didn't have a lot of those. Linus is still at it. He's not retired. Right. So we don't spend a lot of time thinking about the exit plan in our field. If you're a professional athlete, your body will tell you when it's time to go. And so when at CoreOS we reach this peak.

I felt like I've done everything I've ever wanted to do.

We've used your command line tools. I started my career, like some people were not even born when I graduated high school, and they started their careers from those books. And so I felt at the time that I had come full circle and I was starting to think about the exit. Part of that journey, I remember spending time with Jet Propulsion Labs, JPL, part of NASA.

And I remember being there and and I was so excited because the movie The Martian had just wrapped up filming there. And they gave me a tour of the facility, like the Mars rover, the new one before they launched it to Mars.

They were QA and it was just going in a circle around the track. And I'm like, had a little laser on there so I can split rocks and they were showing me how they improved the wheels over time. I went to another lab and there were a bunch of scientists working in there and it looked like a fish tank.

And I described it that was like yeah, we we d determined that if you want to replicate parts of Mars surface and I might be getting this wrong, like the rocks that you find in like a fish tank, we can replicate some of this stuff and maybe slightly different than that. So I'm watching these people work and they show me like how spacecraft has evolved over the last twenty or thirty years. And I'm like, wow, you all seem to have a actual purpose.

For the first time I've seen people using technology not to just make more apps, not to add numbers in a database, but to actually have humanity do something. And so they were not all about Kubernetes and Docker and Python and Go. They were like, We're just trying to get a person to Mars and back again. I remember part of the interview process, their their interview questions are, if you had to deflect a meteor, How would you do it? But it has to hit one state.

So now you're in the leadership position, what would you do? And you're just explaining the answer, like, you know, I would kind of bring up a bunch of experts and then, you know, you gotta think about the ability to evacuate people and you have to explain yourself. And the core part of my answer was You would have to explain yourself almost

Twenty four hour live stream. Here's the trajectory. Giving everyone the countdown. Explain every decision you're making. I chose transparency. I chose truth. I chose like look, we have to deflect it. There's no way to make it zero. So we've chosen this state and this is the evacuation plan and we estimate that this number of people won't make it.

We're just being honest with you. No need for conspiracy theories. It's live. And I was like, wow, what an interview. A twenty year career at that point, never had an interview that made me feel that way. And so I actually was going to go to NASA after Core OS. I even signed the employment agreement. I was gonna move to Pasadena, California, and work at NASA on the Mars mission and lead up the infrastructure and the infrastructure team.

And of course Google called and was like, Hey, come to Google. And at that point I was like, For what? You have hundreds of thousands of employees. I admire Google. I've been there before, but not in that capacity. I was gonna go to the headquarters.

All the things I wanted to achieve in that regard was there. And so I was like, why would I come work there? I'm just gonna be a cog on the wheel. I'm gonna go there. I'm just gonna disappear. And it's gonna make me work on Google stuff all day long. What's the value in that? I've seen the peak of this. And they were like, We won't do that. I was given the opportunity to do Dev Rel, and it's the first time I ever did it. But DevRel represented freedom.

No tickets, no write code, measured against squeeze benchmarks. I was like, I don't want that. I wanted to be able to make impact. And so the team was smart. They were like, look, we've got this area called DevRel and we'll let you define what you do.

I'm a person of impact. And so the first thing I did when I got to Google was like, Where's the customers? How do we make money on cloud? So I need to figure out how to talk to the customers. So hey, where's the sales reps? Hey. If you ever need anyone with Kubernetes expertise, call me. I will fly to Disney. I will go to Walmart on site and I will whiteboard for six hours because that's the revenue component.

So globally, went to Australia, Canada, doesn't matter. We don't even have a region there yet. I'm going to bootstrap it. So now I'm growing my impact on the revenue side.

There's serverless, there's databases, there's metrics, there's there's so many things here. So now I'm like, I need to learn everything. And I want to employ all of my skills. So it turns out my time in financial services. Means I could be an exec sponsor. I knew how to go from hello world to hello revenue.

So if I got into an exact briefing, I didn't waste everyone's time showing them the latest feature of Kubernetes. It doesn't make sense. They want to know how these tools come together to lead to actual outpac impact and outcomes. And so I matured there. And I also got smart. You gotta go to other teams and you read the OKRs. Right. So another team might say, hey, Kelsey, we're really trying to get more adoption on our metric stack.

And I remember the first thing that I started implementing at Google was a thing called empathetic engineering. So you have a lot of smart Googles. These people are brilliant. I mean, extremely brilliant. To the point where the hardest problem Google had, in my opinion, was what to build, not how to build it.

I want you all to install Kubernetes, but you can't use any scripts. And remember these some of them are distinguished engineers or principal engineers. Some of them worked on Borg. Some of them are just the original creators of Kubernetes itself. And it was so fun to watch them struggle because it's like, do we install Docker first? What version of Docker? Can we put this on Ubuntu? Or does it need to be Red Hat?

And so an hour goes by, teams are four are like, nah man, this doesn't work. I was like, great, you all walk and stop. I'm going to show you how I would do it. And of course, I know how to do this because I get to prepare, right? So not a knock against them. I'm just like, all right, Debian, tune the kernel this way, put Docker on there, put etcd, put the API server, put all these things. There you go. That's how you do it.

And I was like, yeah, but I mean of course you had to prepare, of course. And so The question then was from an engineering perspective, how will we make this better? And then people were like, Well, if we had OS packages, this could have been AvGit install and we could have just used local machine. Like, that's a good idea. And someone's like, We will make that happen.

Another person's like, even if you had those packages though, you still need to know what order and where the config files go. So kube admin was born, which was a command line tool that gave you a procedural thing. But the other thing that I remember from my career was I don't want just a tool that abstracts everything from me. I want to know how it works. So I wrote the guide Kubernetes the hard way.

And that guide is what I use to help teach people at GitHub in the early days pre Microsoft how to like run Kubernetes on bare metal and and walk them through that guide. And so it was that empathetic engineering that helped me make a huge impact on cloud because I can go to every team, every org, and instead of guessing what their roadmap should be, giving someone who would spend time in the field.

Given someone that had this enterprise background, hands-on experience across lots of tooling, I knew where people were coming from, I would say Based on where Google sits in the landscape and in the competitive landscape, given what our abilities are and what our customers need, I think this is. But I never said it that way. I would get everyone in the room.

the promotions were a little predictable because I would be able to make impact. Right. Sometimes you build prototypes and sometimes you would contribute to things like cloud functions. But the overall goal though was to use every skill you had. So working with PMs, we gotta add Go support to cloud functions.

We're the team that made Go. Amazon has support in Lambda for Go. We need to do this. And also, I have a keynote at the next Go for Con and I'm talking about serverless. And I have two choices. I have to use Lambda or we can use Go function. And so we sprint, we get it done, we get all the Googler's reality, we get it checked in, it's in Alpha, and I tell the PM, Hey, you wanna go to GopherCon? We'll launch it on stage.

We launch it on stage. We tell people how we designed our worker, how the internals work, and then they can just come sign up. That was the trifecta. You do the work. You have the vision, you execute, you launch, and you land. Land. And the landings compiled over seven years is how you go from like I think maybe L five to Distinguished Engineer.

it's it's a very different you just had landings that that created impact. Like and and you were focusing do I read it correctly that you were not focusing on your promotions or the next next level. You you just wanted to, you know, do the best work that you could.

And then once you get to a level seven, now you kind of pave the way for the other people that want to come up through that icy path. And then level eight isn't the same as level seven. Level eight isn't the same, or level nine isn't the same as level eight. And yes, there is a formula to some of the promotions early. So from three to four, four to five is little formulaic.

You have a ladder, there are things that are expected of you, and the decision making on those type of promotions are localized, meaning your manager, maybe a director. But then outside of that though, when you start to go higher, it's now expanded where there's now other teams that have to understand your impact in a way that can't be biased by a local team. So if you think you've made a lot of impact and then people across the orb do not agree,

that allows you to really throttle what impact means, right? Because if it's just your team, I really like this person. Now everyone is a distinguished engineer. And so I think Google did a really good job of saying, look, it was okay to be like L five or six for your entire career.

And I structure there's examples and you go through all of this stuff and you get feedback. I remember one year I was like, I'm not doing that. This year I'm just gonna talk in like the first person. Hey I'm Kelsey, I work on these things, but not these things.

These things are important to Google. So here's exactly what I did, but more importantly, here's the people I brought along. Here's the teams I've impacted and here's the results of this. So I did this project and here are the results of that. And I'm talking in this way like I'm ignoring

the process. I don't really care about the template. And as someone who at that point in time was on some of the promotion communities where we're looking at the promotion packets and making a decision as a team, I decided to write my packet for those people. So that when they got it'cause m look, if you're having to do a read a lot of these, it's it's hard.

You're like, oh man, they're all so dry. Everyone's being very safe. Everyone's only telling me what I want to hear. I don't even know the person from after reading this whole packet. I said, I'm not gonna do that. I want them to see me as if I was in the room advocating for myself. And I remember getting feedback on that package like Kelsey, this is like come on bro, this is

You know, thing is I was taking it seriously. It wasn't I wasn't making a mockery of it. I just wanted to make sure that they understood what I was doing and I was aware that I wasn't just trying to play the game. I was trying to approach this process authentically. And I got promoted off of that package.

And look, it could just be sometimes the work sometimes speaks for itself. But a lot of times people can't see the work if it's not presented correctly. And so that kind of slingshot at me. And the reason why I tell this story is that Every distinguished engineer doesn't get that the same way. Everyone thinks it's like, oh, how much code did you write? What complex thing did you build?

And for me, I think it was the impact on Google Cloud's culture. The empathetic engineering thing became an official thing that they used to onboard other engineers. It became things that had a whole team behind it. They used to go give them in a product manager that would evolve the program and integrate into HR. Uh the philosophy around other engineers saying, Kelsey, we just did an empathy session. I want to show you the results of it because we're about to ship it.

And then engineers started really thinking about the customer. I mean, Amazon was always known for that to some degree, but to help Google get on the same page. I mean, I'm pretty sure other people had an impact, but there was a direct line of impact from those type of programs and also me diversifying, moving away from Kubernetes into the serverless realm.

moving to the world where you're helping out the Postgres or the Spanner team at a Postgres interface, the Go team, getting a little bit better with cloud, just making other people successful around you. is one of those things that helps you become distinguished. It's the impact, the ability to influence.

At that moment, you are actually worth a hundred and twenty thousand dollars because you now have evidence from the market. The hundred thousand you currently make doesn't look so good anymore. You can't unsee a job offer for one twenty. And so now you have to make a decision. You can commit to that. Leave your job and go make one twenty and start over. Or you can do the ultimatum thing. If you don't pay me one twenty

then I'm leaving. And that puts everybody in a weird predicament because sometimes it doesn't have to be that adversarial.

Sometimes it's just this is evidence that, hey, look, I want to advocate for myself. I know we're not in promotion cycle, but I believe I'm worth one twenty and I would like to have that conversation. And and someone would say, Hey, well you need to go approve it. It's like, Well, I have an offer if you wanna see it, but I would rather be here. And that can turn into a you went to go look for another job, it's like, Oh my God. So I'm

Another engineering team knows that they're gonna need budget for something and they need someone that has a little bit of political capital, a little bit of weight in the organization to help endorse them. So that can be executive sponsorship or you might be assigned to one of the largest custom customers the company has.

But that executive set of duties means that you're going to be making impact above and beyond yourself, typically to support other parts of the organization. So I grew into that at Google. Microsoft was like, we want you to start there.

But my whole career or the majority of it has been rooted in authenticity. I've been working on the things I actually like and care about. This would be one of the first times, maybe outside of some of the enterprise roles. And I'm gonna go work on a set of technology that I wouldn't use on purpose.

And so I went there and I met the Microsoft team and the weird thing though is I had a recruiter and they swapped the recruiter out and it's like, Hey, our mistake. I'm like, What do you mean mistakes? This person's super nice and they asked for a resume and I didn't have a resume'cause it's been seven years

Well it's been actually fifteen years at that point since I ever had to show someone a resume. So I'm like, resume? Oh man, I haven't made one of those in a long time. I'm looking online for a template, like I what's the style of resumes these days? And so I'm like trying to figure out a resume. I'm like, see, this is why I don't waste time interviewing. I don't got time for this anymore.

And they swap the recruiter around and say, hey, sorry about all that. You we don't need no resume. Sorry. Uh that's not this kind of process. What we want to do is make sure that you meet the right people who represent Microsoft.

I was like, okay, that's different. So I get on side, I'm like, okay, what kind of quiz questions are we gonna be prepared for? Because I mean we're not doing quiz, bro. You had committed access to Kubernetes, you wrote the book on Kubernetes, you're leading these things. You helped Star Kube come on, like you have a Wikipedia page, like there's

We don't need to try to figure out whether you made the impact. We can just look at GitHub. There's evidence there. And so I'm meeting all of these leaders. They're bringing all these people who are behind the scenes. I remember me and Scott Gunthry from the first time. And I didn't really know who he was because I just didn't know how I didn't know the lineage and the history of of Microsoft. And you know, of course I'm looking up who these people are and

And I was like, wow, this is they're courting me. Oh, this is what that feels like. I mean Google did it a little bit too, but not like this was Like I'm I'm looking I'm looking at the these are the executive directors of the

And so it was like you can if you want to understand the business, if there's someone you haven't gotten a chance to talk to, just give us a name. And I'm talking to them and the one thing that I really respected about Microsoft was I think by that time they had also acquired GitHub.

And so they had a big vision for themselves, a lot of diversity. And I was like, Okay, there's a lot of opportunity here. They're also all in on Kubernetes. They had just acquired people like Dais and Brendan Burns is there. So I was like, all right, Kelsi, you can come and make an impact. There there's room to grow.

I'm like, all right, I might do it. All right. So I'm glad I did the interview. And I get home, and it felt like that time I doubled my salary. I told you on the way home. And I remember I get this email from Satya, the CEO of Microsoft. And I'm like, man, he wrote this nice email, Kelsey, heard that heard you had a good experience with the team. Remember I did the interview at the Microsoft headquarters, right?

So hey, heard really good things from the team. Just wanted to let you know, you know, you're gonna be respected here. We're gonna support you as a team. I'm like, damn, support as a team coming from the CEO? And the offer was like a PDF, just an attachment. So I read this thing. And so number one, what an honor. This is the CEO of Microsoft. He has so many more important things to be doing than to be emailing me about a role. And I opened a PDF and I has a

Very often in your career there's a zero get added to the equation. And so you're looking at this like, I didn't even know that they do that. We know that it happened. But the person that graduated from high school in nineteen ninety nine that chose the A plus certification didn't know that was available. Even while I was at Google having all the success, and Google paid me pretty well too. But I know you can add another zero still. And so I'm like, whoa, this is

This is crazy. And I'm like, wow. So I um I showed my wife and she was the one that said, you should just go interview, like put your ego to the side. And let's go see what's out there. So shout out to my wife. And so I get the PDF and I'm like, okay, this number is perfect. Honestly, I don't know what to say, but just find out. Like, is this really the only number? So I remember giving a counter. Like, you know what? I think it should be this.

And the funny thing is Microsoft countered back higher. So we're not playing around. I'm like, oh. Now now I understand that I don't understand this part of the game. Yep. And so I have this offer. And I knew that I wasn't gonna go interview there if I wasn't serious about taking it. So I was serious about going to Microsoft. Been almost six years at that time at Google and I went to my manager. I had the same manager, which is legendary at Google. I had the same manager for six years straight.

Even with all the reorgs, same manager, the whole time. He was a director. So even when I got leveled up, they allowed at least someone one level above still to report to a director. And I had a such a good relationship with them, I told him what happened. I said I wasn't looking. And they asked me to come in, so I went in and I'm going to take it. Number one, it would be financially irresponsible to not do this.

So that will be the driving force. And also I get to stretch myself in another way and see if I can make an impact again. He's like, Okay. Um, but it was no ultimatum. It's like I'm leaving. And he's like, I'm just curious what Google would say. I saw no not great, I don't wanna do ultimate.

That's a game changer because he knows me like no other person I've ever worked with. No one knows me as well as Greg does. He knows my strengths, he knows my weaknesses, he knows my ambitions and my motivations. He knows all of them. for him to say that it was important. And so I'm thinking

If Google wanted me to be at that level, they would have done it already. But also pragmatically speaking, no one really knows your situation. Like all these people have thousands of people they manage. They're not targeting you or being mean to you, at least in my case. And so maybe they thought I was making that kind of money already.

Maybe they thought that I was already a distinguished engineer. They how would everyone in the org know? They got other things to worry about. So he presents the thing to them? And I think within hours they're like, this is no problem. Like, hey, here's the you know, don't worry about that number. Here's a bunch of stock and all these things. And I'm looking like, whoa So now I have the money. And I'm at the company I want to be.

A lot of people was like, What does that even mean? Different company, same team. But people were, you know, retweeting it and they liked it. And people just like, oh, it's because of the community stuff and how the different people in the community work together. And that was part of it. But a big part of it was that moment that Microsoft got me the biggest raise in my career at Google. And about three to six months later, I was in San Jose. And Satya was there.

And his admin's like, Hey, um, Sate would like to just, you know, meet me with you. And I'm like, the CEO and Microsoft got time to do anything? So I'm in San Jose and then I go to this hotel and the admin meets me downstairs, like, hey Satis kinda ready to see you now. And I'm going up. And now he know I'm coming, right? Like let's just say the meeting's at one o'clock. He knows I'm coming for like days. And then you go into the hotel room and like the

The doors are open overlooking a mountain range. And Satya's sitting there overlooking the mountain range like a vanity fair photo shoot. And I remember before on on the plane ride there, I'm reading his book, Hit Reframe. And I remember the opening chapter he talks about Starting Microsoft as a developer advocate and now being CEO of Microsoft, advocating for the soul of Microsoft.

'Cause he had been there so long. He saw the birth of the cloud and all these things. So I was like, oh, this and the book was actually a a pretty good read about his trajectory in the industry and his time at Microsoft. So now I'm done with the book. And I'm about to meet him. So I have all this context in my head. So when I walked through, I felt like I had this rapport with him for some reason because I read the book.

And then he's looking out over the moun mountain range and I walked through the door. I was like, Sate, why are you sitting here like this? You knew I was coming, so you're just posing like we're doing a photo shoot. And he just started laughing. And so the tension, at least for me, was way down.

And we had a discussion, I won't repeat everything here, but he said something dope. He said we were sitting around at the table and we asked ourselves, What executive did we want that got away? That means that you still were in the minds of of at least some people. And it's like you were on that list. And uh we had a discussion and he was very transparent. And he said, Um, we gave you a good offer. We think we gave you a good offer. And at that time Thomas Karen had just come from Oracle.

My personal opinion, I like this leadership at Google, but I can understand why some people were afraid of the Oracle DNA being brought to Google. And I think maybe some people in the industry felt like, Oh, this is a moment to go and maybe poach a few people that didn't want to make that transition. And he said something like um we gave you a offer as if you were running away from something, and we should have gave you something to run toward.

And I was like, Damn, that that's poetic And so we wrapped up that meeting and I really felt like, man, I actually belonged to the industry. It wasn't like I'm a Googler or a Kubernetes person, I really felt after that moment that I was an industry person. So I was very comfortable at that point retiring within the next year or so.

I was very conservative in terms of what I was spending. I didn't care about jewelry. I didn't care about buying cars to impress other people. That was gone. I felt free from that kind of thing in society. So the money started to become like freedom tokens. This money means I can get out of the game. And so everyone, at least the people I know or what I attempt to do, you set this number.

And then when you blow past the number, and if you're still a little young, you're like, well, maybe I can change the number. But the thing I was careful not to do was to change the lifestyle. Because I I watch people around me change the lifestyle so you can make a lot of money and still be broke. And I was like, I wanna avoid that because at that point I used to ask questions, why am I doing this?

And for me personally, I felt like I was lying to myself. I love this job. And it's like, no, you learn to love it. And being someone who was good at it. you tried to figure out working around the people you like helps you love it a little more. Working on the things that you're curious in Helps you love it a little more. But you can't deny the pressures of just enterprise, stock price, you know, this person wants it now.

There's a debate on whether we should do this or do that. Um, there's personalities involved. There's parts of it you didn't love and then the other one I thought about was time. Everyone thinks they're gonna probably live to ninety. or a hundred, or we don't even think about it at all.

And when I started zooming out on my career, it's like what are you doing this for? Because once I started having a better answer, when my daughter was born, I'm doing this for her. I'm doing this for my family. I want to make sure that we're all safe and protected. And so I started just changing my attitude. So for example, when my daughter was born, I remember just taking a job where I could work overnight.

I'm just gonna go work in a knock. I don't really care about this job as much in terms of career progression, as long as I can be home with her. So we don't have to do daycare. I'll take my shift. My wife takes her shift. But this was the priority. So then I started to structure my work life around this. Now I was never great at work life balance. I won't a lie. My daughter goes to sleep, I'm back on that computer trying to learn new skills. I probably approached a lot of burnout in my career.

No, yeah, I ain't gonna lie, I'm gonna say like oh and I had a f I did not have that part figured out. And also the thing about burnout, what I think is interesting is if you play professional soccer And you put in a lot of effort and you lose every year. You're gonna feel burnout. I'm doing a lot of work and we never win.

But the teams who actually win play more games than everyone else because they have to play the playoffs, semifinals, they got a win World Cup. And if you win it multiple times in a row, you play way more games than everyone else. But for some reason the champions aren't tired. Yeah.

Let me store these things away because here's what the math means. And I always calculated the math on interest payments, not stock increases. At some point you get away from that. Like I want Vanilla US Treasury bond, what does that pay? So how much money do you need to live off a fraction of this money?

And I started just making the calculations and then you start negotiating salary around these particular things. And you start asking yourself, Am I making an impact that is worthy and deserving of these things? So then of course you get into things like investments, startup stuff, blah blah blah. But then I'm starting to say, Oh, I'm getting close. I can see it now.

But I have to stay the course and be disciplined. And I have to structure my career in a way. So once I got that light at the end of the tunnel, I started making decisions based on that. I'm going to bet to go to CoreOS instead of something that pays way more money. Ah, NASA looked great.

But man, this changes the trajectory over here at Google plus I'm gonna have to step way up to be able to walk on that particular stage. At least that's what I told myself. But also Google is the type of place that could pay for performance at that level. So I was like wow, this is a good opportunity. And so now that I spent all these years thinking about why do you work? I never had a good answer, but I never accepted the lie.

And so I was like, I'm working to be me. And you become a distinguished engineer, but you realize you're a junior person. You didn't put as much work on learning how to live and the relationships and the things that you do when the computer gets turned off. You didn't put any effort into that. Not a lot. Not as much effort as you put into the work.

So it wasn't leave as soon as the conference is over. You'll stay one more day. And I remember hanging out with people like Liz Rice and her team. And they were like, We're going to a bath. I'm like, uh, what's what's a bath? It's like, oh, it's like big swimming pool. And we went to one of these huge parks where there's like you walk in circle. And then you go in the building and there's all these plunges.

And so when I got back home, I didn't talk about the conference, I didn't talk about the keynote, I talked about the escape room we went to and the three hours we spent at the bathhouse.

My wife can sing and she knows the lyrics. So sometimes I'm singing a song. She's like, that's not how that song goes. And like I was what I remember. And now when I listen to music, I actually pull up the lyrics. And I listen while reading the lyrics so I can really understand what the song is about.

And it's little nuanced things like that where I was like, Hey, life doesn't need to be so fast and this is why you see me sometimes online, like, the fact that people are over indexing on productivity doesn't necessarily sit well with me because it's like If you just do productivity, you're gonna miss everything. You're gonna miss the experience. You're gonna miss this part that are hard. You're gonna miss the collaboration with your team.

If you just go through too fast, you're gonna move right past it because you're a human. You're not a computer. You're an actual human and you don't work only for productivity. Maybe that's what your job believes you are. And there's that saying you're not your HR title. And so for me, part of that was like dallying back to like, oh, you're a human. Act like one. So I started to invest in like my relationship.

Talking to people, being patient. Go to the school board meeting. Do some of the offsite things. Spend more time with your child and her games. Teach her how to drive instead of going only to driving school. Cleaning became a big part of my everyday routine. And look, if you have enough money, you can hire someone to clean your house for you. No problem. And there's actually nothing wrong with that.

But boy, I enjoyed the parts where it reminded me of all the success you've made. There are some people who don't have a stove. There's some people who don't have a refrigerator. And so when you would clean them thoroughly, like take everything out, look at all the expired foods, sort everything back out, clean it back to the condition when you bought it, put it all back together. And what I noticed was for the people who had

In my opinion, success, they could afford to do that. It is a luxury to be able to afford to go slow. Because when you're really, really busy, you need an admin because you can't afford to book your own ticket. When you're really busy, you cannot afford to clean your own house. And some people would say, I make way more money having someone else clean my house than I can go make money doing everything. It's like, I promise you money isn't everything. It is not.

And so for me I said, Wow, now I have time'cause some people only have money. Now I had both And I decided that I could actually slow down. And the thing that maybe some people noticed Even my outward projection changed. I was way more methodical. And so the work changed, the keynotes changed, because I started to incorporate the philosophy. I brought the people into the keynote. And I started asking questions like

I know what I want to show them, but then how do I want them to feel? Because I wanted sometimes I wanted people to feel excited. Sometimes I wanted people to feel a little bit embarrassed by the state of our industry and the complexity of we added for no reason. And I noticed that was just starting to be like the full Kelsey was starting to be on display. So then I was like, All right, it's time to retire. Where am I gonna walk out to?

And luckily I was practicing just enough of like who Kelsey is to start doubling down on that as a retire. And I didn't necessarily do a good job. I'm only three years into this. So I'm a junior retired person and I make time for lots of things, but I still want to hold on to all of this knowledge. all these parts of our craft. So I'm still doing advisory. I'm doing investing. I'm still doing public speaking. I don't speak as much about low level technology things.

I do try to put a little bit of philosophy in there, but I know that I'm still holding on to that part of my ego.

It's gonna add a lot of complexity and you're not gonna get what you want. What I think you want is this, let me show you. And as an advisor You're not necessarily the person's boss. You're just trying to give them something that they don't have. So you're advising. Sometimes you become an engineer and you get really close to the executive team where they check with you before they make any big decisions.

So you're an advisor. When you start to advise at a very high level, then you also share the outcomes, right? A lot of software engineers that work at larger tech companies, they have equity. So if you just don't focus on heads down, do your job. And you get into more of those advisory roles, then you start to realize that maybe you start to have a little bit of effect on the stock price.

Right. So my actions, if they turn out well, then we get there. So okay, so how does advisory work in the startup world? When when I first started doing it, it feels like an exact waste of time. Number one, VCs have large amounts of money, so they can invest in a thousand companies. And ideally, one of them will return the entire fund. But when you and I do like in Vangro investing or advisory, typically when I started advising, you would take advisory share.

Ninety nine point nine nine percent of the time they're worth absolutely nothing. Number one, you're gonna get diluted to hell and back. Oh yeah. You don't even know how the taxes work on this thing. You may exercise them in the wrong way and you may end up paying money.

And so when someone say, Hey Kelsey, we want to advise, I say, stop this. When I first started, it was like I need some equity, so advisory shares. But I understood something different now. I said, look. I might need a quarter point, half a point, or a whole point, depending on where you are with your funding rounds. How much risk am I taking giving you this time, and how much impact will I make? Let's say I asked for a quarter point of a company.

And so what I started doing, instead of a four year vesting, I would say look, I need one year, no cliff, ten year exercise window. I'm never losing again. Not on early exploring.

And the retained component, I used to think about them as dividends. Right? So you may give a dollar amount, it could be fifteen hundred, it could be three thousand, it could be five thousand. And you get that every month for that one year. And so in your mind, it's like, all right, I get sixty thousand dollars plus equity. And now it's like, all right, you should probably call me because it's just a very expensive person sitting there. I remember the first time I had a a decent exit.

I advised a company, uh Pixie Labs. They were doing Observability was a small twist. On Kubernetes, they were doing is observability. And what they were doing is leveraging EBPF so that way you didn't really have to add any agents or instruments. So the way EBPF works is you almost get to the kernel level. So if two applications are talking to each other,

at the lower level, I can actually see that networking traffic. I can see what port it's bind to. And I can also see that it's a Go program. And I can even walk the tree like a deburger would. And so they were doing observability this way and they had, you know, I'm almost like, hey, we're going to compete with Datadog or something like that.

And as an advisor, I looked at it and said, Look, uh, you could try that, but most people are really not interested in changing observability stacks just for a slightly different way of doing things, even if it is E B P F like, what do you think it should be? So they're in stealth mode. Their investors are like, yo, it's time to come out of stealth. We gotta start, you know, getting some revenue from this thing. I said, Hey, we need a few more months.

And we need to take another approach. So I'm sitting with the team and I'm like, you know what you have? You have like an agentless thing. And they also have this thing called PixieScript. that allowed you to kind of make your own dashboards or aggregate your own metrics. And I started like, oh, if you're a system administrator, you can wrap them like command line tools and you can like create a thousand clusters with these Pixie Scripts.

And so this idea that we should pivot just a little bit, change the messaging just a little bit before we come out of stealth. And they gave a couple of presentations with this agentless observability. And then we did a keynote or uh we did like a pixie day before coming out of stealth and we show system administrators this new vision. It went well.

I did this opening keynote, I showed the vision, I interviewed the founders and like why did this need to exist? Why not just data dog? The next day they had offers. I think one of them was for maybe being where But the other one's from New Relic and they got acquired by New Relic. And I was like, Wow, what does this mean? And I remember the lawyer came and said, Hey, we need to accelerate all your shares.

And this is the money we are. I was like, whoa, this can work. But also, I felt like I actually made an impact. We did pivot and then the way it works in advisory with VCs, they like returns and word gets around. Kelsey's advisory can have impact. So other founders are coming, VCs are recommending some companies that need help. So over time, you know, some founders reach out, like Guillermo from Vercel reached out.

Like hey Kelsey, this is when they were making their pivot from just being purely serverless and front end to thinking about going a little deeper in the stack. So I spent a couple of years with the Vercel team. Docker can go on and on. And that helps you build out a portfolio. And so I was like, you know what? Stocks are cool. Remember I started to divest from that once I had my retirement plan in place. I was like, but I do like the concept of the entrepreneurial mindset.

And so helping these companies get to the next level, even for my little short amount of time, the little small impact and then being able to share in the outcomes became a major part of my advisory work. So my advice to anyone that wants to do advisory work.

It really helps to be a domain expert deep. So if if a team is about to build out their engineering team and you've been an engineering manager or a team lead, don't just go say, oh, when we were at Google, we did it this way. That's not what a startup needs. What a startup needs is you just say, listen, as you build out your team, you have to think about growth trajectory. You're gonna have to think about vesting schedules, you're gonna have to think about personality types, impactful work.

Junior versus senior spread, when to bring in engineering leadership, when not to. That's the type of advisory that can help accelerate a startup from one stage to another stage. So then that might be your type of advisory role. So when you meet a founder and say, hey, listen, here's my domain expertise. Here's an impact I can have. And if you think that's going to be good for you at this stage,

And don't get offended if your advisory is no longer necessary. Because maybe they're gonna move on to something much different than what you're good at. Let them go get a new set of advisors, because you've done your part.

But the other thing that struck me was with Gen AI as well. You of course is everywhere now, it's impacting. Mm everyone is using it, trying to trying to figure out how to best use it. How have you gone about understanding Gen AI, especially with your approach of like, all right, you know, once once up at a time.

And it's unfortunate that it's th that way, but I really do think about it. And the weird thing is when you say you think about people, people find that very odd. If you're mean, if you're narcissists on Twitter, that's normal. That's expected. If you want to get over people, if you want to game the system, that's normal. Almost expect it.

But when you say, I want to just be kind to other people, that feels weird. People are like, What the hell is this? No, no, no. We're it's we're all just game in this thing. My philosophy around technology really is this people first situation. So when crypto came out, everyone was like, oh, we got these tokens, blah, blah, blah. Kelsey is open source, you know, blockchain. I said, I don't doubt any of these things.

I've been a part of open source movements. I've been a part of things that maybe have threatened other people's jobs. I get it. But this crypto thing, I can't help. Not think about the financial system in a way that impacts real people. Real people are forced to work. Right. So if we go back thousands of years, you can go out into the forest, get something to eat, and that's it. Now you have to get a job because the forest is off limit.

So now we force people into this cycle. This is reality. Now you're coming and saying you want to change the currency. Right? Not all of them, but some of them did. So I'm not as concerned about how blockchain, crypto v that's not as interesting. You're saying you want to change the currency. There have been countries that have gone through currency resets.

This is not a nice experience. Because if you had a little bit of money, a currency reset can mean you have now no money. If you had no money, it may feel like it's impossible to ever get any more money. If you're retired. What do you do during a currency reset? You don't have any way of making new money, you're retired now.

So these are very, very real things that I thought that group of technologists were ignoring because money go up. And so when I would have those conversations with them, they wanted to debate the low levels of like crypto and how transactions are settled and you know, things about encryption in the blockchain. I was like, yo, that stuff is fine and we can go back and forth, but at some point we have to talk about how it impacts actual people. When Gen AI comes out

I'm now super into the philosophy of everything. So there's one part of it that's a little personal. Of course we spend all of our careers learning all these skills, training our own models. Right? We've learned to program yeah, we learn to program

So before we do anything, we all figure out what needs to happen. And then we have to convince the computer to do it. And every keyword, every if statement, every function call is a decision we're making. And of course the syntax kind of gets in the way from time to time. So Stack Overflow, we go. So Gen AI comes out.

And early stages is kind of like people just talking to the machine. And I've never been impressed by talking to a computer. I'd rather talk to real people. So I don't really care too much about that part. Yes, it mimics human capabilities for people that want to talk to a computer. Knock yourselves out. But then we get into the code generation piece. Now we're back to where I was with the crypto stuff. I've used the compiler. It generates a lot of code for me.

I've been doing that for a very long time. I haven't written any machine code. Now post things like, hey, I'm adopting the zero token architecture. If you're like, what's zero token architecture? I was like, instead of burning tokens, you learn things. And you think for yourself and just complete tasks. And they're like, oh, why would you want to do that? It's like, because we taught the machines. I don't know why people skip this step.

Hey, Kelsey is gonna be this artificial intelligence, gonna do all the things. It's like, but we trained it. So all those times I'm writing code, the books I've published the comments back and forth on helping people solve problems, it's all in there. Maybe it's arguable that they have their own worldview based on that, and maybe it's slightly different, but I can never put the machine over a person under any circumstance.

And I think there's a subset. I don't wanna say everyone in the space is doing this. But there is a healthy subset of people who really believe what is the purpose of a person? Why do we need them to write code? Why do we need them to build software? It's like maybe you don't understand what the job has always been. We are trying to solve human problems.

And we use whatever technology is required, in some cases the technology happens to be software. And software ain't required for every human endeavor. And I think a lot of people are just in this bubble where they believe software is the only way to solve any problem. And then they think Gen AI solves all human problems. And this is where I start to push back on that narrative. It's like you learn to love this job.

And you forgot what the rest of the world is doing. And so I feel like some people are now trapped And that person again, I keep going back to that person walking into the industry for the first time. Luckily for me, they were looking for people that had skills and there was a pathway, so many pathways for us. Now the new generation that's coming out, they're unsure of themselves.

Hey, I'm watching the news, you guys keep celebrating people just using Gen AI to do everything. What am I going to do? And I just can't accept the answer being you're just gonna come in here and use Gen AI to do everything and all of you are now just the same.

Let's look at your Amazon bill. Let's look at the architecture. Let's look at GitHub. How do you manage issues? How do you all work together? I want to get a sense for the team, the product, and its trajectory. And when AI is involved, the one thing I just do before the thing kicks off. In this meeting, do not say AI. Because what we don't want to do is use a big umbrella to describe what you're doing. Let's get concrete details. These are computers. These are computer programs.

Yes, just like when I saw a regular expression for the first time, is a different way of thinking about software than, you know, imperative things if else then. So I get that. But now you have to show me what you're actually doing. So when we do that, when I put that handicap in place. Now they're forced to show me the problem they're solving. They don't just say, hey, AI for healthcare. Nope.

Show me exactly what you're doing. And so with that handicap in place, the really good founders, the really good technologists, what they do is they say, hey. Here's a problem. And here how an industry currently solves the problem and here's the drawbacks from that. And since they can't say AI, they can't say ed gentics. They just have to show me how they make the problem better. Now, as a technologist, I know that if I gave you a random PDF,

There is no easy way for you to procedurally program parsing PDFs. You you just can never anticipate everything you will ever see. But it makes sense to use OCR. Or if you don't know what an OCR is, you might say, I will go use cloud to do it. Whatever. Yeah. There's probably an easier way to do this than using a large language model. There are smaller models, smaller techniques. So my advice to them would be, you know, you don't need to use an LM for this.

there are smaller models or smaller AI techniques that are not Gen AI. So they're like, oh my good feedback. We can probably lower costs here. But then when they take these technologies and they do something novel with it. I'm like, you know what? That is a good product for the people doing this work. And I'll say, You see that? You did that without saying AI. So on your website, why are you burying all the value of this platform by putting AI in big layers before we get to the value?

And so maybe they do change the website to actually talk about the value. When they do the demos, they start with the value versus hand wavy text boxes where they put in a prompt and it does something magic. Physicians don't want to do that. They want to see some of the other contextual things and leverage AI to bridge the gap between what they're currently doing and now what's possible.

So my advice to people that I'm advising, you know, all the stars, I gotta make an AI pivot. I said, if you all pivot to AI, then you all will have a problem. It's like kids learning to play soccer. They all run to the ball. No strategy. Spread out, figure where you add value, and play your position. So when I'm advising a startup, what is your position in this big landscape?

You can't all run into the AI ball. You gotta stand back and figure out what which what value you're going to add. So one of the companies I advised are called MASH Driver. They have like a visual kind of infrastructure as code. You take Terraform, you give it some metadata, and then you can interact with it visually.

And part of that visual interaction allows you to do things like this app needs this database. And just drawing that line allows the credentials to flow to the other app under the hood, and now you have a config.

So now Cloud is the big thing. Everyone's like, No, I don't need that. I don't need Terraform. I'm just gonna use Cloud to manage the cloud. Now someone with experience, I'm like, this is about to be real fun. Because I've seen what humans do when you just give them the AWS console. Watch what Cloud's gonna do when you give it the AWS console. And so knowing this, it's like, okay, here's how we can add value to this.

Number one, we can take the things that you have in this visual box and split it up into value props. Number one, in order to show things visually, you have to have context. So let's call it the context engine. And then that context engine can be queried by cloud. So instead of pointing at WS. is now pointed at only the resources you use. And for people to understand why this is important, if you take some of these agents, they just start investigating the console like, woo.

What's Lambda? Nah, don't need that. But Lambda now is now running. What's this law balancer? Oh, don't need that. But now that could be running and you don't even know the mess that it made. Yeah. It doesn't need all the messages made. So now we say guardrails or give it context.

So I said, oh, okay, so instead of doing a full-on pivot in a naive way, let's re-reposition here. So now some of the features we have become guardrails. Some of the things the way MASDR does things with IC or infrastructure's code can become skill. So now if you bring Claude to the scenario, instead of starting from scratch, We can just allow the agent to interact with this platform the same way humans do. And then what we end up with is if a human wants to interact visually, it works.

If you are just like want to have some automation in your pipeline, then just call the APIs and you still can interact with the context and the deployment engines. But if you really just want to use cloud code, Then now Claude can interact with the same guardrails and structure that the rest of your system does. And then Claude becomes a little clearer. We're not asking it to be magic.

We're asking it to be an alternative interface for getting something done. Skills.md becomes an implementation detail. And then when we do a webinar where we present this, just watch the light bulbs go off. for the people watching and for the team

And that to me is the type of advisory where you can look at Gen AI. I'm not just like a Gen AI hater. I just don't like the naive promotion and adoption of it. I think it should be way strategic. And since I think about Gen AI as a tool versus the great human replacement. Then I can use it in a way more pramatic way.

I'm going to assume those will get slightly better over time or human and loop will catch those things. So I'm gonna give it for the sake of this discussion, I'm gonna give it a little grace. The things that I think it should do well that I think don't get used often enough. There's this concept where people really think it makes sense to do inference every single time. For example, if a human writes a piece of code, we'll write it once, right? Let's say I wanted to

authenticate to an endpoint. So I will call the code, I will go look at the documentation, go to Stack Overflow, figure out the example. I'll call it once. If I had to do this twice, it's probably going to become a function that I call multiple times in the code base. If I do this like five to ten times, this may become a library that I import in multiple apps.

So that's tends to be the human flow because there's no need to infer or write from scratch this particular thing. And we've done that. That's where open source comes into play. And when we're really doing a good job, these things get just baked into the framework. And they're just there.

So if you find yourself generating the same blocks of code over and over, even though it feels fast or convenient, it is still insane. Yeah. At some point you should say, whoa Why is cloud generating the same block, copying things everywhere?'Cause we know where this leads.

Create a VM, a network, a storage device connected to a VPC, and then attach credentials. That's not intent-based. I want a VM. So the first thing you kind of see from this movement is things like MCP, where we wrap these imperative calls into an intent-based thing that says create a VM. And then that reflects out to the other ones. But here's the thing, I saw this before that.

I saw this with Kubernetes. Kubernetes does the exact same thing. When you say give me a ingress or a service in Kubernetes, that reflects out to seven calls too to create load balancers, SSL certificates, DNS setup.

Maybe Ruby did a better job than some other languages where they try to give you things like until so that way it was just flowed better as you were writing the code. Other languages are a bit more rigid. It's like, what does Fn mean? So you have to go look it up every single time. So writing code becomes very laborous. So when people write a natural language prompt,

it kind of tells us a lot about the intentions of of querying a database. So I'm hoping developers learn better API design from that front. The other part was I still get frustrated. I want to learn new technology. I go to the website. There's a little bit of documentation, but for some reason developers still write documentations as hints, as clues.

Right. I'm trying to learn a new programming language. I won't throw any of them under the bus. And I go there and I'm just gonna go look at the standard library. First thing I like to do sometimes is just like, I'm gonna parse a JSON file, get a feel for the language. And there's like, there's a JSON library, sweet, we're off to a good start. Click. And you look at the documentation, you just see function calls. I'm like, can I just see in a working example? What do I import?

What do I put in this thing? What comes out? And then maybe what did the errors look like? I just want to see a full example. They're like, no. You just get reference. So then what do we do next? Then we surf the internet. And you might land on Stack Overflow, someone's blog, that they give you a full example. So now I gotta copy and paste it and see what this thing does and hopefully it's up to date with the actual documentation.

This has been the loop we've been going through for so many li years. So when I used things like Clot or various tools. They closed that loop. You get the example right here. And let's not pretend that Stack Overflow examples were perfect. They were not. So to me, like having something where the tool would then try to build it to tell me if this is correct syntax or thinking to make sure that it's giving me a good suggestion, even I can say that is an improvement.

But I hope we learned that maybe we should not just give hints in the official documentation. I'm watching people write these huge markdown files to give the agent context. How about you write documentation to give me context so I can have fully working exams?

Because to me, again, software development is to me a human endeavor assisted by tools. And so there's a style to documentation. There's a personality to documentation. It's like ri like a movie, like you're trying to educate a person. I don't want just hints of hey, this thing exists for these reasons. We conform to this particular specification. There are multiple ways to write this code. Here's the most popular way.

Here's what bad code looks like. Here's what high performance code looks like. Here's when to use this library. Here's when not to use this library. I want that kind of in depth as I'm training my own model. And what I'm seeing now is, which I think is a good thing, people are writing a lot more documentation to be consumed by the agent to give it context. I was like, man, I wish.

we had the same motivation just a decade ago because I think a lot of us would have been way more productive if we didn't have to try to do a wild goose chase every time.

Uh, which is something writing code used to take a lot of effort. It took a lot long time to uh be good at it. Now with code reviews, there's now more tools coming in and some people are like worried, like, okay, like what is happening to my profession? Like the craft of writing code seems to be something that we can offload and more and more people are offloading including

a prominent people, what will this do to software engineering? And what advice would you give to people who are experienced software engineers? They're a little bit worried'cause it's it's a big shift. They still wanna, you know, like be the whatever great engineer will look like in the future, but

What steps might be able to take? And especially I I especially like your your take because you're I think you're pretty grounded in just looking at this from a from a vantage point that some of us are not.

the radio, the calculator, the compass, all of these tools people used to buy individually. The top thirty of those electronics from the last forty years are all in your iPhone, all of them. That means some electronic makers have gone out of business. They're gone. You did that. Not in a malicious way, but you were part of that.

And so the software developer has been glorified for a very long time. The internet, some people would say, caused the downfall of magazines and newspapers because of the convenience of having a software approach. And so you have been part of the change to other industries and other people yourself. What did you think about that? Did you even think about it at all?

So let's not be surprised if you find no sympathy from all the other professions that you've helped force change upon. So I think that's step one, you have to go do that reflection. Because if you don't do that reflection You won't know how to behave now. You're gonna be complaining and people are gonna look at you crazy because where was this empathy before? You might be very excited about this. And not realize you're only excited because you're in position to benefit from this.

So if you work at an anthropic, of course this is the future because it's in your hands. If you're at NVIDIA, of course this is the future, because you will be selling the picks and shovels. And so you gotta ask yourself, why am I excited? Now, what I don't want you to do is necessarily feel guilty about it, but I need you to see the big, big picture. It's gonna help frame everything else.

The second thing I want you to do is ask yourself, what was my job? Remember, there was a point in my career where I was lying to myself. I thought my job was to be the less best Linux administrator ever. You as a software developer, you may have thought your job was to be the only person in an organization that can write code, and since no one else could do it, you are safe.

Right. And so you didn't learn any other skills, networking, product management, design, talking to customers. Nope. All you had to do was write code and you were set. And you probably made more money than everybody. And you were fine with that. Now you got caught. The only thing you were good at. is now been commoditized. And again, you did this to others.

So let's say the vision you have for yourself is only in this very narrow realm. You're going to be very afraid of this trajectory because all you know is software developers write code. That's it. Some software developers still don't write text. Still don't know how to deploy anything. And so they are really afraid because they can't see any other way that this plays out.

Now, if you're a full stack engineer, you're probably like, Man, there's so much more than just writing code. You have to do architecture, you have to do design, you have to do so many other things that I love Cloud because now I can focus on those things and I can use these tools instead. So I can see why that person would have that perspective.

Now, I understand why that full stack person has a perspective of watching the same people commentate that the code generation piece replaces everything else. They're gonna be like, no, you don't know what this job is. It's way more than just writing code. Writing code is the last step. If you're a security engineer, you're probably like, we never figured out security for the pace of the current enterprise.

Just go slow. A lot of attacks are I'm about to board a flight. And let's say you're an admin or you're a VP and the CEO texts you right now we need to wire the money to Oracle to pay for the license. They're gonna cut it off. Right now, this needs to be done immediately. You look at your phone. It's definitely from the phone number of the CEO. You have a good relationship. Everything looks right and it's moving fast. So you're like, man, I'm on a ten hour flight.

I need to do this now. Turns out the attacker knows you're about to abort this flight. They've seen all your previous text messages. They know how your manager talks to you. They know that you've moved fast in the past. And so they now are primed to get you to do the exact same behavior again. And you could be the VP of security. So you shouldn't know better. And sometimes that naive confidence will make you feel like I'm obviously not being fish.

I've done this many times. This is definitely the CEO who would know how we actually operate and who would know that I can actually do that. So what do you do? You'd make the transfer. And just like that, ten million has been wired to the wrong place because you moved fast. It wasn't because you were not smart. It wasn't because you were not productive, because in this case you were productive, but you did the wrong thing. So when I think about code.

There is value in having a healthy pace. Let's say you're an insurance company. You sell insurance. Hey, make model. How old are you? Have you had any accidents? Okay. Here is your insurance for the year. Simple. Very simple thing. If you're an insurance company, that's all you are, you're kind of close to being. Now you could say, well Jin Ay, we should get into payment. We should compete with DoorDash, right? We have all these tools. Let's go and

What schema? Should we really collect someone's social security number? Or should we avoid it? Yeah, I can write code to parse a social security number really fast. Like, no, no, no. Should you even do it? And so when you write code it almost makes you slow down again.'Cause there's been times where I thought I had a good design. That's that phrase, writing is thinking. So is writing code. So as you're writing the code, you're like, hey, wait a minute.

This loop is ridiculous, right? Not only is it gonna make the computer warm, this is not the right thing to do. There's a better data structure than the algorithm that I'm using. So then you stop and say, hey. The data structure's wrong. We need to change the way we print receipts on the cash register. Sure, I can write this code, but this is the wrong data structure. While I can generate the code, doing reports are going to be a nightmare.

Summarizing this data downstream is going to be a nightmare. Stop everything. Now that I've thought about it, we need to change the architecture from the top down. So decision making sometimes does benefit from slow. And when I'm saying slow here, we're not talking waterfall six months. Yeah, no. We're just talking about maybe one more day before you go at it. And I think some of us are going to miss that part because Clot spit it out. Ship it.

And then I remember when I was now five or plus years of experience and stack overflow started to become big and I was like, nah, you don't want to go to Stack Overflow'cause you're not gonna, you know, learn the the real thing. So but now what The current old guard is saying, which is, you know, where I'm I I guess I I'm part of it, is like, well, if you use AI, you're going to miss

Learning the basics. And when you have learned the basics, it's so much easier to use AI. And I wonder if we're just repeating the same, the same mistake as the previous ones did, which is the new generation usually figures out the tools. They understand how to do it. Or are we rightly concerned that some people who are

Coming into this AI native, they're now learning to code. They they can jump through so many layers that they will just not, you know, see what's under, understand, or are we just like making assumptions that might not be true?

And so they got really far with that. Now for the what they're trying to do in accomplishments life, they'll probably be fine. But let's just say you are a software engineer. And the idea behind software engineer is not limited to just producing apps. Software is the interface between hardware and things people wanna do. So there's a whole bunch of things you need to learn. So if you wanna be that type of software engineer, you've gotta learn hardware too.

If you don't understand hardware, you can never work at that level. And look, if that's not your job, then so be it. But you will never have that creativity. I remember seeing someone was like, hey, you can do isolation without a VM. I was like, how would you do that? He's like, oh, because when you boot the kernel, there's a thing you can do before the kernel loads to isolate it in a way that you can lock down processes. The only reason why this person knows is they know the full boot sequence.

From firmware to switching to the kernel and the tricks you can do in between. Now for me that doesn't work at that depth. I'm thinking there's only virtualization, CPU isolation, things like G visor where you intercept system calls, but never did I think about the boot sequence. And so yes, you can get very far, but as someone like we applaud every version of Opus that's released or Chat GPT, but there are versions of yourself that get deeper from these new trainings.

So no, you don't have to. But if you ever want to get better at anything, and sometimes that depth, that nuance. is the thing that leads to an invention. Right? If you know how a compiler works, if you know how memory management works, that might give you enough information to say, oh, I can make a new programming language. If only thing you know is the surface, you can't even imagine.

how you can create another programming language that is better fit for the task at hand because you've never gone that deep. I'm not saying everyone wants to do that. So I think it is fair to say All I wanna do is come in, get a job, and if that job can be done by using AI tools, I think the side effect of that is, then that job will be commoditized. It has to. That's just the way it's gonna go.

But I've always seen myself for my entire career, I want to learn more. I want to go deeper. I want to go so deep that I can create. And I think a lot of people who are doing this, the reason why we're having this reaction, some of us, some of us, part of our careers have been the creation part. There is no spec for this. There's no protocol for this. But we're gonna make it work. A lot of people that are doing like the reverse engineering and the hacking, there's like

There's no framework for what I'm about to do. I just know how memory works and I don't care what your security tool does, I will make it do what I want it to do. You need to go way below the surface. And so for a lot of us that are saying this, we know the value of the fundamentals that lead to the other stuff.

And so if you tell them next generation, oh, you don't need to learn these things, it's like, that may be right in the short term, but we know for a fact your career will be limited. And that may not be a problem. You have to decide. But make no mistake, if we put this much effort in training the model so that it can spit things out, you better make sure that you are willing to train your own model. So my advice to people would be

And maybe we should talk about it different. Maybe we shouldn't have so much fear mongering around it. Maybe we wouldn't should put it a versus this. We should just say great artists tend to know how to mix colors. And it is in your benefit to understand the primary colors so you can mix them to get the other color.

And it's a superpower, right? So you don't have to go buy, you know, imagine an artist trying to go buy sixteen million colors and put them on the desk because they don't know how to mix colors. If you teach a person how to mix colors, like you can get any color you want. And I think that's the way we have to approach it. It's just another skill that if you had it, you might just unlock some creativity. So I encourage you to learn it.

and still ended up as distinguished engineer at Google Cloud, only a few hundred people who hold a title at the company. And he retired at the top three years ago when he decided that he no longer needs to work for others. What an inspiration. One thing I took notes on was when Kelsey said how every job is an interview. When Kelsey was giving that Gophercon talk and PXE booting CoroS from his slide deck, he had no idea that the CoroS team was sitting in the audience.

And that's how he ended up at CoreOS. When he was contributing to Puppet at nights and weekends, he didn't know that James Turnbull would walk into his office one day and recognize his name. He just kept showing up and doing the work in public. It's a lesson worth remembering. Do the best work you can at work. It might unknowingly be your job interview for your next step in your career.

I also found the Microsoft offer fascinating. Kelsey did not use the offer from Microsoft as an ultimatum at Google, even though he could have. He just told his manager the truth. truth. And then Google mashed the offer. Obviously at this high of a level, there's no universal composition negotiating advice that always works. Being a straight shooter with high integrity is something that is good to keep in mind. I was also inspired by Kelsey's focus on minimalism.

He treats money as freedom tokens and made sure that his lifestyle never inflated with his salary so that early retirement was always a Not just fantasy. Finally, Kelsey's AI takes. His point is grounded and pragmatic. AI does not change what software engineering is actually for. The job was never just to write code. The job was and is to solve human problems. The engineers who understand this.

are going to be fine. Do check out the show notes below for related the pragmatic engineer deep dives on Kubernetes and other related topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks and see you in the next video