Mitchell Hashimoto’s new way of writing code

If AI agents can write code, open pull requests and ship features. Do we even need open source contributors anymore? Michelle Hashimoto, the co founder of HashiCorp, has been thinking deeply about this, the future of open source. And how to efficiently integrate AI into his day-to-day workflow. Mitchell built the tools that power modern cloud infrastructure, Terraform, and the HashiStack. He also created a popular terminal, Ghosting.

And I consider him to be one of the most thoughtful voices in the industry on how AI is changing the craft of software engineering. In today's episode, we cover the original story of Hasher Core. A failed university research project, a notebook of unsolved problems, and an email from his future co-founder that he answered in two minutes. His honest, unfiltered take on working with AWS Azure and Google Cloud.

both the arrogance and also the brilliant engineers who never thought about the business. How he's adapted to AI coding tools, why he always keeps an agent running in the background, and his practical advice for engineers who have not yet worn rubs AI agents.

And many more. If you're interested to hear from one of the most hands-on builders in the industry and want to know where AI tools are useful versus not, then this episode is for you. This episode is presented by Statsig, the Unified Platform for Fly. analytics experiments and more. Check out the show notes to learn more about them and our other season sponsors Sonar and WorkOS. Be here in person. Yeah, it's it's cool to meet you in person after so many years of following you.

You've had such a massive impact on on the tech industry, on software engineers. But how did it start? I think the high level is the same story as a lot of people. I self taught uh around twelve, thirteen, early teens, motivated by video games, same like same as a lot of people. Um, although I really quickly realized that I liked web. You know, web was new, Google wasn't out yet I think web was new, and so I I kinda like really quick I I never

became a video game programmer. I really quickly just became a web programmer, PHP. um Pearl, that sort of stuff. And uh because I was so young, the only way I could learn was through whatever code was published online. And so that's how I got acquainted with open source. I I didn't know that's what it was called then, but a a kid with no job, no money, um, parents

didn't want to buy, you know, uh professional books are were like I don't know what they are now, but they were like fifty bucks then, right? And And so they were like, No way. Right. This and also they didn't believe I was gonna read it. And so there was no way they're gonna buy that. So, um yeah, I'll uh just anything I find online was my my in into coding. I'd walk to school every day with a group of friends.

There's a period of time where I printed out the first or second chapter of the PHP manual. I I remember it was about thirty to forty pages of of paper. And I never programmed so all the stuff and M twelve is is very confusing. So I read the whole forty pages every walk to school. And I don't remember how long it took me, but I did that a long time before, you know, I remember this one moment where I was walking to school where suddenly I understood

what these dollar sign things were. I like it like for whatever reason it just came in and variables. Yeah, yeah. And I I really understood no, I never heard that word before. Like you don't hear the word variable. as a twelve year old out in any context. And finally at one point it like hit me that they store things and things could change and I remember just like

weeks of reading this thing and not understanding it, getting the school so excited thing like it it triggered. And then after that I remember stuff happened really quickly. What what kind of stuff did you build? Websites? Yeah, websites. It was gaming related websites. It was like a lot of like game cheat stuff, forum, software. Yeah, I mean I had a lot of fun

cloning websites? You know, in a poorly, but like uh PayPal was out er than then and I really wondered like how does money get transferred over the internet? How does that work? So I tried to build like copies of cloning websites. I did like masquerade as a eighteen year old on um uh like freelance websites. And so I got, you know, a hundred bucks here, fifty bucks here to do like image, like upload stuff. I decided to study computer science in college. Um went to the University of Washington.

I mean I guess that's when you would call it serious, but I was I was like really I mean, I was coding every day as much as I could through high school. Oh, okay. Impressive. Were you alone with this in your friend group. There were there other people doing it or was it kinda lonely? It was lonely. It was very lonely. It was it was lonely in the real world and then I quickly found online friends through like MSN Messenger and AL L and C Messenger and

forums. I found online friends which many I I have met now and I still keep in touch, which was cool. But no, I mean like Back then, I mean being a being a programmer one no one knew that word, but but being into computers was like a social death kiss. And so

Uh, even my closest friends didn't know. My best friends and stuff, like I hid it from all of them and I didn't talk about it at school and stuff like that. So it was just a secret until I went to college. And college is when I decided to like let it all out. The big like break that I got. was I blogged and uh my freshman year, late freshman year, heading into summer, um, after it of college, someone just emailed me out of the blue and I kinda thought it was a scam. It was just like

Do you wanna you know, it is do you wanna be a Ruby on Rails programmer? And I didn't know Ruby. I I was a PhD programmer. Um I had never done Ruby, I'd never done Rails, but I got this email and I'd never been like headhunted before. Like I didn't know what this was. I was also eighteen, so I didn't really know what to think about it. I probably would have not responded except that the person contacting me was in LA.

And so I did respond and we set up a meeting, like a real physical meeting, and I m met'em and met the company and realized this is real and they're serious and genuine and I took that job and Uh yeah, yeah. I mean that was I learned a lot on the job there. So that was a huge change. Um was it a startup or small company, something like that? No, it was a consultancy. So it's kind of like one of those standards like this is like two thousand seven Ruby on Rails was

had blown up. It was already very popular and uh There's all these consultancies that ex uh that appeared on nowhere that was basically like, We'll build your minimum viable product in yeah. And we're one of those shops. So great job for a college student because we'd see a client for like two months and I would build a

YouTube style website and then I would build like a philanthropy website and then I'd build an e-commerce website. And like it was just like I got to learn all these different technologies and different scale challenges and different like and there wasn't a lot of scale'cause we're building MVPs, but different like thinking of scale problems. Um yeah, it was it was great.

How did eventually HashiCorp start? Or what what what happened between like getting getting this this Ruby job to a few years later? It kind of starts with this Ruby job. Um there was one guy that worked at the the company and and he's he's pretty into his privacy, so I won't share his name, but he was my boss and there was no

Roku, there was no engineer. So you had to like self host and Ruby on Rails hosting then was kinda like difficult. So he was the guy who got all these projects hosted on on dedicated servers. And I didn't know anything about that. And I and he ran Linux and he had long black hair and he like didn't use a mouse and all these things that were so weird to me. And I was just intrigued. I just he sat in the corner, he didn't want to talk to anybody.

Um and I just want to know more about what that world was. And Luckily, despite appearances, he's very nice. And so, um yeah, I I I think as soon as I showed a genuine interest, started asking a lot of questions, he started just giving me

challenges. He's like, Well, the first challenge I remember he did is he unplugged my mouse. And it's funny'cause I I don't think Yeah, there's an era of time where if you did that it probably would have been some kind of harassment or something. But he he literally said unplugg he'll unplug my mouse and said, You're never gonna work with a mouse again. So figure it out. I'm not gonna tell you how, just To unplug my mouse, restart the computer.

your problem now and took the mouse away. Mm-hmm. Uh took me about a week and I got really good with the keyboard. Harsh lessons. Harsh lesson. And once I got good with the keyboard, um he said, Okay, here's

He he installed screen on my you know, early TMUX. He installed screen in my terminal and said, figure this out. You're gonna use this now. You know, there's no questions, like you will use this. And and he just slowly instilled on it um on me and As we got there, then it became, you know, here's SSH, here's

a package manager. He's like he slowly taught me more and more and that got me just in I loved inf like immediately. It was like this is super cool, super fun. So that long winded process got me into infrastructure. And then it simultaneously or very shortly afterwards I joined a research project at the University of Washington called the Seattle Project, which is a terrible name'cause you can't Google it. Um

But it's called the Seattle Project. It was. I I'm sure it doesn't exist anymore. And it was again another popular thing during this time was uh kinda like um folding at home. It was this I they were trying to generalize folding at home, which is can a bunch of people compute of different, you know, it could be your home machine, it could be a unused rack, it could be

in your basement, it could be around the world, but can you compu can you donate all this heterogeneous hardware and then can you generalize a scheduler on top of it so that academic institutions across the world could just run workloads. And not just like Not just research, like the job I got was basically to very vaguely to create Not the scheduler component, but like

create the ability to spin up all these nodes um and and a and a bunch of other stuff. It's it's very vague, but but it was this infrastructure problem. And I completely failed at it. Like I I tried for a quarter, but um from a technical side I just failed. And And I wrote down on his notebook like what I thought the pieces were missing that I couldn't have to do.

solve this problem in a quarter in a 10 week period like why? Well we need this, we need this, we need this. It's interesting to see how structured Michelle was in his approach in defining components that would later become parts of the hashy stack. And this leads us nicely to our season sponsor, WorkaWhat. One thing I've learned from studying great engineers, Michelle included, is that they're very deliberate about what they choose to build. Great engineers.

don't just ship fast. They think in systems. They understand leverage. And they're careful about what becomes part of their long-term service area. If you're building SaaS, especially an AI product, authentication and enterprise identity can quietly turn into a long-term investment. SAMLED cases, directory sync, audit logs, and all the things enterprise customers expect. WorkOS provides these building blocks as infrastructure.

So, your team can stay focused on what actually differentiates your product. Great engineers know what not to build. If identity is one of those things for you, visit workwise.com. And with this, let's get back to Michelle's notebook with all the components he would end up building at HashiCorp. And I still have this notebook um at my house here, but the problems are really like

you know, I have no way to declaratively manage the different resources that are out there. I have no way to network these together in a private network. Um, you know, I I wrote these things down. And there was a lot of stuff there that I never ended up building, but a subset of that was ultimately what Hash Gorp would end up building. And I shared this with my undergraduate

like boss who is Armon, who's who was my co founder. Yep. So he was later later became your co founder. Yes. He was the my my boss on the undergrad side. And I shared it with them as kind of an exit interview. Like, this is what it is. And then some period of time passed, not much, weeks passed and he emailed me out of the blue and was like

Do you want to do a startup together? That you know, you're a teenager and you have no idea what this commitment is. Well you you're like twenty one or something at this point. Uh probably not even. Probably probably nineteen or twenty. Yeah. And he emailed me out of the blue, it's like do you want to do a startup? Like n person you never met.

Or you barely met, uh never met personally, like all this stuff. It's so funny. And he emailed me that at like eleven thirty near college. I emailed him back in two minutes and said, Sure. And he remembers thinking, Wow, yours bought it so fast that he's just in. He's ready to go. That was sort of the start of our friendship. And then uh And uh again, like there's overlapping pieces here, but I was also at the time working on something called Vagrant.

And Vagrant was, you know, came out of the consultancy less the less the research project. It was solving the problem in this consultancy where we had new clients every two months and we had different teams. How do we create reproducible dev environments so I could go help somebody without a lot of billable hours? So so this is a development environment that you could spin up quickly, right? Yeah, yeah, yeah. The metaphor I always had was

I didn't use Windows then but the metaphor I always used was how could I double click and open a dev? Yep. That was a metaphor I used because it's a good one. Yeah, what what what the problem we're having was any hour waste in a consultancy that you can't bill is just a waste. And so it was basically like, if somebody else is behind schedule, how can I jump in, help implement a feature and jump out? And we were in that era

just setting up the dev environment for a project might take you half a day. And you can build that for the client, right? The client will only pay for the work. Yeah, you can build that for the client. So it'd be like four hours of work wasted. And it would probably mess up your dev environment for your app.

virtual client because it would be a different Ruby version, a different Rails version. And so you would kind of destroy both ends. And so Vagrant came out of that, which was I just need to go over there. And what ended up becoming Vagrant Up, sweet.

you know, few minutes. Let's help you for the next two hours and then And how how did you build it back then? Was it some kind of virtual machine or Yeah, as with virtual box, virtu uh Oracle. Well it wasn't it was Sun then, but um VirtualBox and and that's that's another cool constraint, which is that I was a college student, so I had no money. So This was expensive back then, right? Uh virtualization was excess expensive, but VirtualBox was free and open source.

I don't care about the open source side. Um, for that I was never gonna read it, but yeah, it was free. That was why I did it and And that's why I did that and not like EC two which should come out by then. But I didn't do EC two because I I didn't have money to pay for these instances. So um yeah, that's that was the constraints and and I like bringing that up because I think

so much of software engineering is understanding constraints and working with these constraints. And your prior podcasts there were, you know, called the forces, like static and dynamic forces. It it's that and And I think that helps create better software, um, when you have constraints. And that was my constraint. So yeah. So that was We have Vagrant, we have this failed infrastructure project. Um we have my boss at the consultancy getting me into infrastructure.

And all of the and then I mean externally we had the cloud being introduced. AWS. I w I went to school University of Washington, so I was right there. Right in the epicenter of it. Uh Amazon was next door, right? Amazon very next door. They donated a bunch of credits. right away I knew about the launch. Um most of the C S students at UW interned at at Amazon, not necessarily AWS, but also including AWS, but all over Armand, uh, interned at uh AWS and so

Like I I I was in the bubble of like cloud, cloud, cloud, AWS, AWS, AWS when people were pronouncing S three like S cubed. Like people didn't know how to pronounce it right. That's how that's how new it was. And so yeah, all this stuff kinda came together. And uh kind of led me on the path to to build tooling to better manage it. And at that moment in time, when you saw cloud, you know, you saw it was being big.

Did you know or have a conviction that it would be big or as big as cloud has become?'Cause this was I'm just trying to put yourself back. Like this was very, very new back then, right? Totally. Yeah. And and I think, you know, like if if if I imagine I assume more people would have been skeptics for a thing that is just a fat or whatever. What what was it like? Can you can you bring us back a little bit there? Compared to the d to the d to today.

it was very unpolished, I guess is how to describe it. You know, like EC two was I mean AWS in general was very unreliable. Um S three was o the only ever reliable piece. Everything else was was totally unreliable. Um and there was only a few services like

EBS didn't even exist when we started. So there was no durable storage besides S3 when when I first started with it. It just felt very raw. Um and I don't f I don't I never really viewed it as this is gonna be big. I mean eventually I thought it was gonna be big.

what I viewed it as is this is the better way to do it. This feels like the better way to do it, just yeah, uh at a base level. Like whether this wins or loses in the realm of markets and social like popularity, I don't know, but this felt good. And so that that's what kind of pushed me towards it is And I I say this over and over I'm I'm really motivated by like what's the most fun and what like feels right. And that it just felt right to me. Um I think where I started making the bet

me and Armand both started making some kind of bet was not just when we started Hoshcorp, but we started Hoshcorp on the basis of like multi cloud. And I really like to like contextualize that at the time we were starting this, which was like two thousand one and two thousand twelve, which is that AWS was huge.

Azure didn't really exist and Google Cloud didn't really exist. There was Google App Engine, right? It wasn't even cloud. I I used to use that when it was App Engine, yeah. Yeah, yeah. And so in that context, as we were pitching these cloud agnostic tools, I mean we got a lot of raised eyebrows being like, this is a waste of time because AWS is the only player in town. And our conviction was at that point Cloud is gonna be huge and anything that's economically huge.

other people want a piece of that pie. And so you're not gonna just have AWS. You're it'll be huge, but you're gonna have these others pop up and Microsoft is not gonna sleep on it and Google's not gonna sleep on it and who knows who else and Who knows? And that was our conviction. That was our bet.

And uh it mostly played out that way. So when you decided to start Hashi Corp, uh you had Vagrant, like was the idea to like, you know, like invest and commercialized Vagrant and did you go out to raise money or did you start doing it with, you know, bootstrap? How how did that go?

It wasn't a commercialized vagrant. So what we had done is Armand and I both worked at this mobile ads company startup. Um there's like less than thirty people. And we had built like with Python and C, like these really um rough prototypes of these ideas that I had in this notebook of like service discovery and um like an early version of Terraform we called Launchy. We had DNS based or service discovery service discovery by

connecting an off the shelf DNS server with Postgres and we did all these like hacky things, but they felt good. Um and again we get like get back to this like how things feel to me to motivate me. Like it felt right, directionally right. I Graduated the the the environment

in Seattle was not very start up heavy at the time. It wa basically everyone was like, Are you gonna work for Amazon or are you gonna work for Microsoft? Yeah. That was like kind of the and and like to a certain extent Facebook was starting to show up up there, but that was it. I knew I wanted to work for startups, so I had I I moved to to San Francisco. So I moved to San Francisco, found a startup that would hire me, which was a mobile ad.

thing um and uh just wanted to learn. So that that's the short step there. So I ended up in San Francisco. Um and I convinced Armand was actually gonna do a PhD at Berkeley and he was accepted and in and he was This is a huge deal. Huge deal. I mean a incredible program. Um and so he was gonna go there and he would have done amazing things there. But

I convinced him to join this mobile ads startup. He actually took a year to ferment on the PhD. He's like, I'll give it a year. Yeah. I'll join this mobile ads. And I'll go back for per for sure. If it doesn't work out I'm gonna go back. And what ended up happening in that year is is now what we get to. Um which is that we had this these these this hodgepodge of protot prototype tools that felt right.

And we were going to all these little start up mingling parties. You know, it's like things like GitHub drink ups, but also just like our This is such a San Francisco thing and that's why I think it's even though I don't want to live there again, it was so magical at the time, um, was like across the street was this company that was called Zim Ride at the time, ultimately came Lyft and they invited us over

to get drinks and have pizza to demo this new app with a mustache that like didn't have a name. And this is like stuff like that. You were there when I was born. Yeah, yeah, yeah. And like that happens All the time. Like all the time in San Francisco and and I'm it's not unique to me at all. Like yeah. There's there's a bunch of stories there that I think aren't worth going into. It's just like it's fun. But I went to all these things and people would just talk they're all

Bunch of tech guys. Right. And and you'd be like, What what are you working on? And and w th there's two things I realize. One is all these companies are cloud first. They're all just adopting AWS first. There is no there was no D dedicated. This was like in in twenty eleven, twenty twelve or so like like they they just like went and paid for paid for cloud w which was brand new, right? The previous generation just had had on prem I remember Pete Butts and and

server admins, they had roles for those, all all that jazz. That was just gone. That must have been a massive show. The only one is very Twitter. Yeah, yeah, but I I think you know like we probably have to emphasize that that was this was a massive shift in the industry, right? And it probably was was only happening in Silicon Valley or like um probably.

Yeah. Well we'll have it of everyone else. At a scale that was larger than anywhere else it was probably in Silicon Valley. The joke used to be because AWS is so unreliable, the joke used to be that when AWS went down, uh all these startups finally became more cash flow neutral. And they would lose less money. Um, so there would be like a huge, you know, US East outage and and everyone would be like, Are you gonna migrate regions? Like, no, we're saving money right now But

Yeah, getting back to it. Uh everyone is cloud first, cloud born, cloud native, whatever you want to call it. And uh the other thing was they were hitting all the same challenges that we were hitting and

they didn't use our tools'cause they were just like internal prototype tools, but but I knew that our tools felt good. So I had these two things come together where I had some ego, some hubris where I'm like, I'm pretty sure we're building the right thing, along with I think the industry's moving in that direction and like we could cut we could come together. Um and so that led to let's start a company based around that. The fact that I had Vagrant was more of like a industry respect

I mean Vagrant wasn't that big then, so that that's not saying much. Um but it was it I just had some foundation publicly with To give some credibility. to head in this direction. Um that was about it and we we started Hoshcore. And then what when you decided you incorporated, you know, got the things. Did you decide to raise money?'Cause again back then I guess it wasn't as

common wisdom, you know, why combinator was probably starting around that time. So like startups were startups a big thing or was it a given that okay, if you start a startup you're gonna raise money? In in my c social bubble it was pretty much a given. Um and and not j not just that. So I we incorporated um I self funded um I I I

transferred twenty thousand dollars from my savings account into this corporate account initial funding. Um and I worked off of that. I didn't I paid myself zero dollars for the first six months. So th the twenty thousand was purely towards whatever things the company needed. That was the first six months. And then Armand joined after six months. Um and

And we decide to raise. Uh and the motivation there really is there weren't many there weren't many other options. There were basically three options as I saw it then, which was uh bootstrapping, um, right? Just like build something.

S make money and as it becomes affordable continue to grow, reinvest and grow. Bootstrapping. VC on the other side. And then in the middle was like what I called patronage, which was not not like not not Patreon style stuff today. Like that infrastructure didn't exist. There was no subscriber donate type infrastructure than um patronage was more like you might be able to convince a company like VMware.

to pay your salary for you to work on some idea. And the best example is Redis at VMware. And yeah, and we kinda laid out this plan that we wanted to do, um, which was which at inception of the company included Terraform, console. No, it included everything but Vault. Uh Vault came a little bit later and we looked at that and said, if we bootstrap this, even if we hit it out of the park.

This is gonna take us like a decade just to like build the software. And that's in the best case scenario. This is just gonna be slow. And and the problem with slow is that things have a window and and the cloud is growing so fast that if we were that slow, someone else was gonna do it their own way. I mean that was I guess that was the primary issue is we really just wanted to go fast. You need you knew you needed to. Yeah. I need to we need to hire

many engineers right away and start building right away. And so V C was the route we chose. Can you talk us through the the first several products and and what they do? You know, we we know Vagrant, but just for those who are less aware of of what what became the hashy stack later, right? Yeah. Let me see if I can still get these in order, I'm pretty sure Um so this Vagrant was predated it. The first product that came out of Hoshcore itself was a product called Packer. Um kind of understated.

publicly but kinda underpins a lot of things in the industry to this day. That's an image building uh tool. So building Amazon images, VMware images. uh, etcetera. Um I'm not even sure how much like publicly came out, but there are whole cloud like multi billion dollar cloud platforms that all of their official images are like the service images are built with Packer.

Everyone was trying to utilize this horizontal scaling, auto-scaling nature of AWS. That was the dream. And if you were it's kind of like the uh what is it, cold star problem with serverless today. If you were waiting tens of minutes for your server to be ready, you couldn't react.

Um and so my idea was Do that, snapshot the image, and then next time just spin up that image. Um, and so that was Packer. That was Packer. So Vagrant Packer. Um, the next one that came out was console. Um console was uh solving the networking problem and not networking, it was more solving the service discovery problem, which was

you have all these machines coming and going before, again, like to contextualize this, before you would have a static set of machines that had IPs and you would probably use DNS or something, but the IPs didn't change that much. So you could be like, oh, my database is here and it's not moving. But if you're in this world where

web servers and load balancers and databases are just breathing. They you know they're that's how I always describe as breathing. They're their creation, destruction, creation, destruction, like constantly.

then things are happening at a scale where the service discovery needs to be much faster. Um and not just f faster, but you want to be have better guarantees that when you get a response that, oh, it's at this IP address, so that IP address is like ready. It's not just, yeah, I I think this is also kind of more mainstream with like Kubernetes readiness checks and health checks and things like that. It was it was bringing that to more like

physical server or cloud servers, virtual machines and things like that. And so that was console. Then after that, I think we did Terraform. Um Terraform uh spins up infrastructure's code, describe your infrastructures. In in AWS parlance it was things like all the attachments to your EBS volumes, gateways. VPCs, subnets, and like connecting them all together. Like the idea was I wanted to have an empty AWS account or any cloud account, and I wanted to have this tech.

And I wanted to say make this text reality, and that's what Terraform is. And and you would wait whatever amount of time it took AWS and you would blink and you would have thousands of resources. And then you with one command again, you could just tear it down to zero. That was Terraform. So that came out. Two thousand fourteen. Um so that was a next thing. Uh and then was Vault Yep. Um vault is is is the easiest to describe as secrets management. Yeah at its core. Secrets management encryption.

grew to do a lot more things for that was it's like like well we have like our on your local level of a machine you have your like your environment variables and doing that at scale at a team level at a m company level at a service services need to access all these stuff securely. Yeah, it was much more focused on the like the the production environment secrets. Um

I had dreams and visions of really solving the developer secret problem, but Vault really never never did that well. Michelle just talked about secrets management. which turned out to be a pretty important focus area for him. In general, security is both very valuable, but also pretty hard to do well. This leads us nicely to our season sponsor, Sonar.

Looking at where we are today, we've now moved past tap completion into the era of agentic AI. Autonomous agents are opening pull requests. One big question. How do we get the speed of AI without inheriting a mountain of risk? Sonar, the makers of Sonar Cube, has a really clear way of framing this, vibe then verify. The vibe part is about innovation, giving your teams and your AI agents the freedom to build and iterate at high velocity.

The verified part is the essential automated guardrail. As agents start contributing more of our code base, independent verification that checks every line. human or machine generated, against your quality and security standards is more critical than ever before. Helping developers and organizational leaders get the most out of AI while ensuring quality, security, and maintainability is one of the main themes of the upcoming Sonar Sun.

This isn't just a user conference. It's where devs, platform engineers, and engineering leaders are coming together to share practical strategies for this new era. I'm excited to share that I'll be speaking there as well. If you're trying to figure out how to adopt AI without sacrificing gold quality, join us at the Solar Summit. To see the agenda and register for the free virtual event on March the third, head to sonarsource.com slash pragmatic slash sonar summit.

And with this, let's get back to HashiCorp and why the company decided to raise six months after founding. But yeah, it it's just basically like we are where do you store your secrets and and the secrets were not just I I forgot the words I used to describe this, but secrets were not just like passwords, but it was also like PII. So How do you protect emails and addresses and stuff for your customers?

Or credit card numbers numbers. Um so vault was core to all of that and continu continues to be. That that is a heart to build, s something like that. Yeah, we were really scared when we built that actually, because um we kind of hid the fact, we never lied about it. But nobody on the team that Bill Ball.

had more than one quarter of security undergraduate security experience. There was no professional security engineers from industry. There's no professional security academics. And uh yeah, we built it. We got a lot of audits because of that. Like we were scared. So we did get a couple we

For us it was very expensive as a startup. We paid a couple firms tens of thousands of dollars for Vault Zero point one to audit it. We paid two. Um we got we shared the early beta with a lot of people who were security experts. in order to review it, not publicly, just privately. Um we got a lot of good feedback. Um, but yeah, we we didn't want that exposed in a sense. So Yeah, I I I understand, but I mean it kind of validates that you can build good stuff with

I guess people who might not have the experience, but I guess people were learning, right? Like Yeah. The security stuff ended up we know, we really quickly hired professionals that helped the product and and the security stuff was always pretty solid. Um but But I think what it really showed was what the security industry needed was a shift in user experience more than a shift in like what it did.'Cause like what we were doing was not fundamentally different than

existing multi hundred million billion dollar companies that already existed, but the experience, the way it you interface with it was dramatically different. And that was I think a good example of that. Yeah. An appravault came. Nomad. Nomad. Yeah. Nomad, which was our scheduler, which was a couple years later. For f to the market. Yeah. Well was you say scheduler. I always described it as scheduling.

Simple thing. You have a pool of compute. It finally solved that problem that I had in undergraduate. You have a pool of compute, you have an app that has a certain set of requirements and it needs to find a place to run it. Yeah, yeah. The the undergraduate problem we talked about. And as you're building out like these uh you said like some of these took years, like how

did the business like Hashakorp as a business work? Like did you did you start to generate some There was no business. Th there was no so so like all right, tell tell me about this one. Yeah, I I think we waited too long to to develop a business, but um for four years there was

There was actually revenue from a couple random sources, but there was no real reproducible growing business. So you were just building this vision of of the the su you know, you're your the the the founder's vision of like, right, we need all these

things that would have taken like a decade bootstrap, let's build it. Build it in five years and figure it out. That was literally it. Yeah, that was literally it and and you know, it was it was all open source and I always had this mentality which was

Which was like, if the company fails, it doesn't matter because if they're good ideas, the open source community will just continue. Um, and so I don't think I would ever tell that to my investors at that time. But, you know, I had this idea which was like the technology was the most important thing to get out into the world.

Um the business. I really sure hope we could figure it out. But It's not the most important thing. And for those engineers who are thinking of becoming founders. Or, you know, m might might be founders. How did this work with your investors? You know, when they put in money, like did they get some boardsies? Did you have to you manage manage expectations?'Cause I kind of I'm hearing just putting a bit of my business hat on is like, you know, for four years you're building these cool things.

you don't exactly have a business plan. How did that work? Or or they just believe that eventually you guys will figure it out or or they sell some kind of traction with like open source? It's traction. And I I don't think what we did was atypical for Silicon Valley. So the really broad hand wavy way I like to describe it is

You know, your seed is about building the product. You don't even know if there's product market fit. You're just guessing. Not you're you're making an educated guess, but you're building something, getting the A. You've sort of proven hints of product market fit, but you definitely don't have it yet. You've proven hints. And then when you get the B, you've proven product market fit and now you haven't really proven like

repeatable revenue. You've you now have hints of revenue. But y you know the product is useful. You know people like the product and want to use the product and maybe want to pay for the product, but you don't know exactly how to get everybody to pay for the product. And then C D and so on is just continue to build the repeatable revenue machine. And so with that framework in mind, we were on the right track. It was basically like build the product.

Um, we had clear product market fit by the A. Um, in terms of the open source, right? We had millions of downloads, a lot of stars on GitHub, all sorts of signals that showed that This was resonating, and

we had zero revenue. And so, you know, it was raise money and slowly slowly get closer and closer to solving the business problem. And I think we were just a year or two late or like later than the average startup. But the general keyframes were the same, just on the slightly wrong timeline, I guess.

And then when you decided to do a business, this was you already had the the the hashy stack and then you built managed offerings, I remember. Yeah. Our first foray into commercialization was a total failure. It was this I really Yeah, we had this product that some people you had you would have to have been a diehard like Hashikore product fan to know this. But we had this first product that was called Atlas and the idea was

Commercially shipping the vision of running all the products. And so that you know, there's a couple death knells there. Um one of them was that you had to run all the products. And so if you were just like a vault user.

you had a really impossible time buying or buying into our commercial product. And the second was just that it was just a huge problem to like attach onto. And regardless of the adoption required, you're trying to solve the problem that multiple different buying organizations in a company

we're fighting over. So like even the people who had adopted all our tools, we ran into the problem of who pays for it. It wasn't as simple as engineering paying for it. Correct. And I I think um one of the lessons that I would have you know, I would have for engineers that become founders that don't have a business background and one of the tough lessons I had to learn is that Companies want to pay for software, but they will fight over whose budget.

owns that budgets are important, right? So the budget has to exist. And if it looks like a networking problem, they're gonna say, oh, the networking should pay for that. So they so I have more budget to buy my other toys that I want. Aaron Ross Powell Or I can hire more people or yeah it could get broken down into like vendor budgets. It could already be earmarked for external purchase. But yeah. So we have this product that was like does security before it does networking before it does

um infrastructure pla pay for it, like does dev tooling pay for it? Like where does this go? And it's just that Spider Man meme where everyone's pointing at each other. Ultimately you don't sell anything. And so that was a failure for that reason. So I don't remember the total time we chase this down, but uh we had a board meeting for sure on a Friday and bright board meetings were usually on Fridays and and we had this board meeting we're based in the city of San Francisco.

Um board meetings were an hour south and in um s real Silicon Valley. And it didn't go well. It wasn't there was no no yelling, there was nobody saying, You guys are messing up. There was nothing like that. It was just the way I describe it is when your parents aren't happy with you, but they don't have to say that they're not happy with you. You know. But you know they're not happy with you. We had this board meeting. We drove home. Armand and I complete drive home was silent.

And nor Friday it's Friday night, so usually what we do is we go straight to our mom lived in the city and I lived in LA already, but we'd go straight back to where our mom's place. And just like have a glass of wine, debrief, talk through things. And we didn't talk in this car right home. Armand drove straight to the office.

I didn't question that. Uh we went into the office, um, sat at a table m not much larger than this. The only difference was there would be a whiteboard here. I I think one of us at that point said Well, that didn't go well. Uh We both knew it. We didn't feel good. And uh the the sequence of events here is now very fuzzy, but at a certain point we decided Let's play this experiment where if there was no sunk cost,

If we were starting from scratch, what would we do differently today? We whiteboarded all this stuff out. What we whiteboarded out was per product, enterprise products, and doing Vault first and all this stuff. We wrote it out. Spent some amount of time there. It's still Friday. It might be Saturday in terms of the time of day, but it's still Friday. I think it was Armand who looked at the board and goes, Why don't we just do that? Like, why not? Like and and and I was like, Yeah, why not?

So we decided that over the course of that weekend So just throw it all away. Just throw everything we were doing before away. We had two paying customers. We're like, just breach contra I don't know, like figure it out. Like get out of it. We're done. And we convened an all hands meeting on Monday.

probably only about twenty, thirty people in the company at the time, but we convened an all hands meeting over Zoom. Um I think and we might not have used Zoom then, but whatever video chat. And we said, Okay, we're switching directions. We are now Enterprise as our customer, open core per product. We would have this open source and we would have a forked version internally that had closed source features. It was a fork, but yeah. Um open core business model. Armand and I thought

People would quit. Like we thought we would lose like We'd have an exact number. We thought it would uh shatter some level of confidence and like, wow, these these these guys have no idea what they're doing. Uh we didn't have any idea what we were doing. Um and You know, yeah, open core even then had a bit of a like icky taste in people's mouth. And so like we thought people would just like philosophically quit being like, No, I came here to work on open source, I'm not gonna do open core.

Um, enterprise was kind of just like a suity, boring thing. There was like so multiple facets of why people might quit. Nobody quit. Uh the vibes. in Slack were amazing. Super positive. Oh. What what happened? Do you think like m why? People had we asked about it in one on ones and follow ups. We asked about it in

It was really like everyone was kind of just like buzzing that we had a clear direction and a conviction. And, you know, there was fear of the unknown, but but before there was this feeling of like we're just We're just throwing darts at the wall and doing this thing and We don't know who exactly who our customer is. And there was all this uncertainty in a different way.

And now it was like, we don't know if this will work, but at least we're just gonna sprint towards this like there's these clear things which was like, definitely enterprise, definitely open core, definitely vault. Like all these things were set in stone that gave us s a different set of certainty that suddenly the company was like Let's go. Um so yeah, nobody quit. It went super well and um we started I don't the t I don't know the time of year, but it was it was like in the fall.

We built Vault Enterprise um by the new year within like the first quarter of of trying to do sales. We could just like tell that it was different. It wasn't like obviously successful yet, but just the caliber of conversation. We're having the the distance we're getting in the buying process a a and the speed we're doing it.

It just felt different. And what was different of this approach? Yeah, I mean part of it just comes down to like the classic startup like listen to your customer and we we should have listened from the beginning'cause'cause

uh our potential customers were s were screaming at us to do what we ended up doing, which is we would give these pitches about adopt all the products and buy this pie in the sky thing and and there were so many meetings where someone would be like, Okay, I'll think about that but

How do you replicate your secrets involved? You know? They would just like ask these questions where if you if I was just listening, I was so blinded. A lot of us are blinded, but I was so blinded. If I was just listening, I'd be like, wait, a lot of people were asking about secrets replicated. And that's a at scale problem. Maybe we could close source that, right? Like

That's what we ended up doing. Is it that was our first feature with with secrets replication. Uh not even across data centers. The first feature was just like a cluster of vault servers in a single uh region. You would sell this. more focused product, but now kind of the problem that I talked about earlier, security was definitely the buyer. There was an obvious budget, obvious person you were talking to. There was a feature that uh it resonated with that scale.

And so we were just having much higher quality meetings in terms of uh getting this done. Michelle just talked about how HashiCorp managed to build a product that enterprise customers cared about and wanted to buy because it resonated with their scale. This brings us nicely to our presenting partner for the season, Statse.

Statsig offers engineering teams the tooling for experimentation and feature flagging that used to require years of internal work to build and is especially important at enterprise scale. Here's what it looks in practice. You ship a change behind a feature gate and roll it out gradually, say to 1% or 10% of users at first. You watch what happens. Not just did it crash, but what did it do to the metrics you care about? Conversion, retention, error rate, late in the road.

If something is off, you turn it off quickly. If it's trending the right way, you keep rolling it forward. And the key is that the measurement is part of the workflow. You're not switching between three tools and trying to match up segments and dashboards after the fact. Feature flags, experiments, and analytics are in one place.

using the same underlying user assignments and data. This is why teams at companies like Notion, Brexit and Atlashin use Statsig. StatsIg has a generous free tier to get started, and pro pricing for teams starts at$150 per month. To learn more and get a 30-day enterprise trial, go to statsig.com slash pragmatic. And with this, let's get back to the episode and what came after they built Vault. And I get asked on the open source side all the time, but these buyers like

corporate buyers do not care at all about open source. They don't care at all. Like they need a commercial agreement. And so the the closed source nature of it, like some people needed Like legal protections around like Code escrow. in in terms of downtime and stuff like that. That was about the extent of it. Otherwise they were like, you know, we need support. We need proof of concept to prove it works. Um we need some white papers in terms of like other customers' scale, blah blah blah.

Um and yeah, that's what we had to build up after that and get going. And then so you started selling at Vault and then you did it for the other products as well, right? Yeah, we did Terraform and we did console. Um

We y we had it for all the products but but you know, all this data's public. You could look at it in in the well, for a period of time it was public. You could look at in like the public reports of when Hoshcorp was a public company. Um, you know, it really broke down to Voltaire form. One thing I I remember is Terraform just became so, so, so popular across the industry. So like, you know, like there's a hashy stack, but I

I only layer that all the other parts exist'cause like Terraform just seemed to be everywhere. Wha wh why do you think that Sudden popularity was It's so funny to hear that because I I I accept and know that now and I feel the same way that you feel now that Terraform is this huge thing. But for the longest time, like we were the vagrant.

Like all the other tools were like no one knew the other tools. And not only that, like Terraform uh I I one of the things that kind of frustrates me, I haven't heard it recently, but for a period of time One of the things that frustrated me was like, oh, they they only won'cause they were first to market. Th I hear that a lot and we were like seven.

The market. Okay. So like to market in in in what category? In terms of that infrastructure's codes per spent a few years. So so there were like other like players who you So many, yeah, yeah. And and no one was a clear winner. It was a warring market, but like That first year, twenty fourteen, when we came out to Air Form, I you know, at at that time one of my marketing strategies was I was at every conference. I I went I traveled an obscene amount

I was speaking wherever I could, but even if I couldn't speak I was going just to talk to people. Um and there was actually a little anecdote here, it was when the covet lockdowns happened in March twenty twenty. my wife and I had nothing to do and I we didn't have kids yet and we opened up our calendars and we realized that It was a f we had been dating since twenty twelve.

in the first time in and almost ten years of our relationship that I will have been in the same place longer than eight days. No. For for almost ten it was it nine years. For nine years straight I had been somewhere different at least every eight days. That's how much you traveled. That's how much I traveled, yeah. And I know there's consultants that travel a lot more and stuff, but like

I was traveling a lot, I was coding a lot, I was like doing all these things. Uh you must have like coded that while while you traveled as well. When I started traveling, in flight wi fi didn't exist. Yeah. Yeah. Exactly. Even now it's kinda patchy. Yeah, so I wrote these scripts that end up Iterating on but mostly used, um, where I downloaded all the GitHub issues and I categorized them and I would just cr break it down into tasks that none took more than ten to fifteen minutes.

And I just created this list and and when I was on the plane I would just one by one bust them out. Uh there's no internet, so just commit'em locally. Yeah. And then I would get back and and some people used to notice this because I would land and you would get this push and people would get these email notifications where like thirty issues were closed all at once. Wow. But I found the key was

pre planning what issues you were gonna work on. I did that o online on the ground. Yeah. And then breaking them down into fifteen minute chunks because I found it was really hard to get into like multi hour even when I was traveling to like Japan or something. It's really hard to get into like multi hour

flow on an airplane. So I was like, I'm only gonna work on the stuff that isn't like heavy design work, none of that. It's just like bug fixes, right? Like just cleaning stuff up. And so that was my process. In twenty twenty one, Hashi Corp went public. What is it like to go public, both in terms of preparing for it? How did it feel? What changed after on the prep side?

I don't have the full answer because I also stepped down from the executive team about maybe a six months before we went public. So I was part of some of the planning and obviously I was very aware that we were planning to go public and but um like for example I wasn't part of the road show or any of that. But Yeah, you know, from my seat, the the parts that I was part of, the parts I had visibility onto, I mean it

it it takes over a year t to do it. So there's a lot of prep and and there's some funny things that you do. Like you do you start running like a public company at least two quarters before your public. Um I don't remember what the drop dead date is, but there's a date where you could just like

cancel going public. And it's pretty close. Like it's like very close to to when you actually like have that day. So you you kinda run like a public company and to the point where you do mock earnings calls. Like you actually with a conference room table, your investors are the public investors that aren't in the room. They go somewhere else and they talk over the speakerphone and ask you the types of questions Um your CFO or VP of Finance gives the fool.

report of the quarter. They try to frame the types of questions you get and you run it and you try to figure out like whether it's running well uh enough, I guess. And that's sort of what the prep feels like. And there's a s uh an obscene amount of secrecy because um

from a regulation standpoint, you can't talk about any of this. And so I mean, you could look back at at even the dumb stuff like hacker news comments. Like I just want radio it it's the clearest signal that a company's gonna go public because I want radio silent on every topic because everything became questionable. I remember there's just a point'cause I I there was a Hacker News comment I gave like eight months before it went public and our general counsel

like in the middle of the night, it was like you have to delete that. After he talked to me I was like, I could see how that might affect things but like I didn't realize it doesn't matter and I ended up deleting it. And is is this because you're not supposed to give public information away or something like that. I don't remember the exact regulation, to be honest. Yeah, yeah, but there there's some regulation about like uh Like not bleaking It's not information or something.

I mean it is it's all information, but it's more about like you can't influence the market. uh in any way. And so yeah, and and you can't make promises because if you say, Oh, we're gonna go public, it might cause even private funding to froth up and it's it's a form of fraud. Um, so yeah, uh basically like I just stopped talking about everything. I don't know how seriously other people take it, but I took it to the point where

I planned this trip to New York to go public and I invited my parents and I didn't tell my parents why we were going to New York. And I just told them And why you go to New York? It's really, really important. It has to do with Hashi Corp. And they were like, Sure. And I said, I can't tell you about it. And they're said, Sure. And I told them maybe a month in advance. We had a dog and we had to get our dog sat.

by my aunt and I just told them we're going on a family vacation up to the point we left. They like I didn't tell nobody except my parents knew basically. Um none of my friends, nothing. Um, except the friends that worked the company. Um, but yeah, it it that's what it's like leading up to it. Yeah, I I was at Uber when we went public and and previously I read that uh while before going public, uh Hashi Corp.

VMware made an offer earlier. That that was like in super early. We went probably like ten years in the company. So so like when when they tried to to to buy you, like what was it like? Did you almost sell at some point? W was there any point where where you were c close to potentially selling?

It fell close and and and I got a lot of accounts afterwards that it was very close. It it came down to like one vote on the VMware board, is what I heard. About two years in the company we were only three employees, me including me and Armand. So we had one employee, I guess. It was two founders and employees, three of us We got approached by VMware. Um you know I didn't know what this would be like. And And it is not what it what it isn't is they don't show up and say, We would like to buy

No. No. That would be too obvious. The way it happens is you get an email from some low level business development person that wants to just like talk. vaguely. And the vague talk is they're not interested in buying you.

large companies is just to have an understanding of the ecosystem. So it's really just like, let's have an understanding. They might have had an executive tell him or her to go talk to this company. There might already be an executive kind of poking around. But yeah, so it kind of starts out that way.

It turns into Would you like to come by your offices and meet in person? Um, oh R V P of Engineering swung by. Let's talk to him. Nice to meet you. Blah blah turned that uh then I think this is the our actual timeline. And then I think uh there was a dinner where there was three VMware executives at the dinner. Um at that point

We thought they might be interested, but it was still so Wow so much dancing. Oh, it is not even this is just this is months before there was even an offer. It was still so social. Like we drank We talked about our hobbies and interests and very not about I mean very basic about like tech It's really more of a vibe. So they'd go to dinner. Uh and then it started to get more serious. We spent more time in Palo Alto at the VMware offices where we started talking about partnerships.

about how how can VMware help our products more? And it starts about partnerships and then it turns into like hypothetical if you had the resources of VMware, what what would you do? You know We're like six meetings in at this point. There's no no offer of anything. And then at a certain point, um, honestly, we were getting tired of it.

'Cause nothing was happening. Any sound like and then you're a startup and you're going to all these meetings that's going to be a little bit more than Oh, and I don't even live in the Bay Area. So I was flying up all the time. It was a waste of time. And um and to a lot of

founders that is the warning I give them is M A becomes a waste of time. So I have another M A merge and acquisitions. Yeah merge and acquisitions becomes a waste of time. So I'll I'll tell you another anecdote after this. But um ultimately we kind of politely had the like, Okay, let's like shit or get off the pot kind of conversation and they uh put an L O I in front of us, which is a uh letter of

Intent. Letter of intent. The L O I was it was one page. Um, you know, it is it's basically like a non like semi binding promise that we're pursuing buying you. No number on there. It's just like kind of vague. Still no number. Yeah. Well verbally. And that's at that point verbally we had gotten a drop of twenty million dollars.

Which doesn't sound that much. Well yeah, but we're well well I'm twenty three years old. Oh yeah, you're the three of you twenty three years old. Twenty three years old. Me and Armand together own seventy percent of the company. Um Okay, yeah. Yeah, i it you know, it it sounds interesting to say the least. What I tell people is you start you start

thinking about the things you will buy. Is you start that's the that's the it's a dangerous path. That's the that's what happens. And we had advice from people who said it's like phenomenally too low, like wildly too low. So go ask much higher. And we asked I don't remember anymore, but we asked for maybe like forty or fifty or something and they just said yes. They said okay. And then you know it's like way too low.

And uh and that was verbal too, so there was nothing binding about that. Yes. It was just it was m it wasn't and it wasn't like yes. It was more like okay, we'll work on that. You know, but very positive. Yeah. Like in this indirect, it's an indirect business sense. Yeah. And it turned into come meet the CEO of EMR. You know, like clearly they're interested'cause we're like climbing still. Armand and I we kinda started getting cold feet because it's it's

The way we described it is it's a dream killing amount of money. It's like you would take the money But you're too small to be important to a company like VMware. So they're gonna just because e even though it's like so much money.

But personally, it's so much more than it's not. But but you know that at the Vuanware level, I guess you see the revenue, they're all that, you realize that for them it's not a big It's meaningless to them. Yeah. It's meaningless. That's crazy. That's that messes with your mind, you know? Yeah, yeah. So it becomes a thing where it's like

personally your life could change. But this thing that we both were truly passionate about, like the thing I wanted to work on more than anything else would end, in a sense, because you know, I would probably get thrown into like working on ESX or something, you know. And you would get a manager of a VM where not even the CEO. The executives make it sound like they're gonna do all this stuff with your products, but like that's just one executive and a cog of corporate machinery.

So we started getting cold feebing like. If they're interested, maybe we're on to something. if we're onto something, do we want to sell out early and sell out in a way where our dream dies? That's why it's like a dream killer. Armand very maturely And he's two years younger than me, so he's twenty one at this time. No, he's he sounds like the older one. Yeah, yeah, yeah, yeah, yeah. He's very mature and uh Armand very maturely

came up with the uh I forgot where it comes from, but the the risk minimization f uh not risk, um the um regret mini gr minimization framework. He was like what Personally on your own, go think and I'll do the same. And let's come up with a number that If we walked in the next day and

And they said, We're killing everything. You're gonna go work on ESX for the next four years because we had a we were gonna have a lock up no matter what. You're gonna work next four years that we would be like, Cool. This was worth it. Like what's the minimum no regret or minimum regret? We came back and I don't remember w exactly our numbers were, but they were pretty close and we ended up at a hundred.

And we and so we're like, it felt so wrong. Like, how could we possibly ask for a hundred? But we're like, we said this is what we're gonna do, and we stuck to it. So we went back. We asked for a hundred. And it wasn't a no. And they they wasn't a yes. This one had a lot more hesitance. It was a lot more like uh we'll get back to you. Right? Like, I don't know. But it wasn't a no. And basically they came back to us and said, This requires board approval.

So we're convening a board meeting next week, like unplanned. Bo that's not when they're board meeting. Like we're convening the VMware board. We're gonna vote on this. And then we heard that uh the vote didn't pass. That was that. It's just crazy how So s such small things could you know, like like like like influence if if that was an extra yes. Who knows what your story

That's hard to but you know, in VMware you might have been clogging away on like this this project. Yeah, yeah. I mean we didn't build Terraform yet. So Terraform Terraform might have not probably probably never would have existed. High confidence, I know who the vote was. I know why they voted that way. Like I know a lot more details, but it's like I it's it worked out obviously my favorite, but yeah. So

you've you've left HashiCorp and and you're you're independent and one thing about cool about being independent is you're just very honest about stuff. And there was this really interesting thread where on on Twitter, you wrote about you said, like, ask me anything about the big cloud providers because at HashiCorp you work with all of them. What was your experience back then?

uh of, you know, like a Azure, AWS, Google Cloud, like like your kind of honest view of how they work back then and possibly like how has your views changed on them? The precursor to that is while I was at Hoshcopio, I I always say I had to be very careful about what I said about any of the cloud fighters because we're partners with all of them, we're partners and I didn't want to insult anyone and and so I was just very professional.

about all their relationships and then like w we like all of them, like Yeah. Or just say nothing. Or just say nothing. You have nothing nice to say, don't say anything at all. And then I left and I was still I kept that up because it was too close. I was I was still flying too close to the sun, as they say.

And then enough time passed where I was like, eh, like my opinion doesn't really matter. And um yeah, so my uh to answer your question, um my broad view of all of them was that AWS was really arrogant. uh annoyingly arrogant was how I'd describe it. And then when you say arrogant, like, can you help us understand like

how you work with them or what part of them or like is it is just it's general? Yeah. Uh I'll I'll start disclaiming this though, that, you know, we worked with so many people there that there were Individuals and all of them who are awesome and nice and kind. And so I'm not trying to make like individual judgments here. It was just more of like how all of it came together and how it felt as a as a whole. So by arrogant I mean it always felt like

they were doing us a favor at every turn in terms of partnerships, in terms of just getting a meeting with them. It always felt like you should be thankful that we're spending time talking to you. And not just that, but also like there was always this subtle vibe of like, we will just spin up a product and kill your company. You know, it it felt that no one ever said that.

Um well it it kinda got to a point where it was sort of like if we don't come to terms we're gonna build this service. It it did kinda come to that. But you know, we did see that later on with Elastic and Oh, that had already happened though. Oh, it happened already. Yeah, just not with us, but with other people. Yeah, and they always publicly spun it as like

Oh, it's so great and and builds the builds the ecosystem larger and we're doing it by the letter of the license and you know, all has truth elements to it. But it's still not a nice thing. No, I I I think like um I I I don't think people paying attention to open source appreciated what Amazon did with

with a lot it it it really hurt Elastic's business and it showed how open source can be weaponized against a company that spends, you know, their blood sweat and tears. And I guess you know, Hashakor, you had the same thing, right?'Cause'cause you you were publishing permissive well, I mean open source needs to be permissive.

So it was MIT or MPL licensed. So yeah. So like Amazon could have spun up any anything they wanted. Yeah. There was like a two year period where I I think for the entire two years, the entire leadership team was terrified that at any moment there would be like a vault service or something uh would pop up.

Um and so yeah, that's that's sort of my characterization of AWS. It really took like, for example, teeth ringing to get them to help with the AWS Terraform provider. Um we had I don't remember the exact number, but we had something like five p full time engineers employed working on only the AWS provider for Terraform, which, you know, maths out full benefits and everything to like a million dollars a year. Um and all of that was

pure open source, pure integration with a commercial entity, and they were not helping us at all. And and they were the last of any of the cloud providers to to provide any sort of help there. And it It came down to some drama where we went to a meeting and basically said that we're gonna publicly say that the AWS provider is deprecated.

and we're done. Like the community could pick it up or whatever, but we're not we're gonna Yeah,'cause you didn't get any help from them. Yeah, and it's taking up too much work and there's too many bugs and you're shipping honestly, AWS is shipping features too fast and like it's just like not worth it. And that freaked him out. Then finally they start helping. You know, they might

recount their side of things differently, but that's pretty much it felt like no movement for years, and we said that and movement started happening really fast. So yeah, there was that.

Um Microsoft? I w uh I have the most positive view on Microsoft. They had a really hairy technical product is how I describe it. It was very difficult to use and Azure. Azure and a lot of nouns like like principles and I d I didn't I still to this day and I've integrated with the servers don't fully understand the IAM hierarchy of Azure um

I just kinda bolted it and got it working with a team and and that was that. But so technically kinda uh but from the business side, super competent, um, professionals and team players. That was like how I describe it. They we We went into every meeting with them and A lot of our meetings, the first question was, how do we both win? That was like the first question and and yeah. Very pleasant. Awesome. They were the first people.

to jump on board uh supporting Terraform. Sure, that's some kind of bias, but like they were consistent throughout the years. So positive on Microsoft. Um and Google Cloud, you know, my my or yeah, Google Cloud in general, it was always like the best technology, the most incredible technology and architectural thinking. And I swear none of them, it felt like none of them cared or thought about the business at all. It was like every partnership meeting we'd spend hours talking about the coolest

edge cases and scalability and how this is gonna work. And like I I think the the best public example that you could just see in history was they were the only company that when they partnered with us to write the provider, they spent a lot of time building this very good I think they called it magic something. They they they fully automated the whole thing. So when they shipped the new Google Cloud thing, it had a Terraform provider resource right away. And not just like

it didn't feel automated. It felt very ergonomic and like it was good. It was really good. And so that they had that. But whenever we would get into how do we do Co-sale? How do we like attribute your sales engineer's quota to selling like infrastructure that's spun up by terraform. Like how do like how do we do this? So like the business side of things. Crickets. Like impossible to get anyone. Not just impossible, it was like even if you got someone, they would

say something for twenty minutes and be like, Okay, cool. We have two more hours. Let's figure this other thing out. And yeah, it was it that's what it felt like. So uh and the other disclaimer I give is all this knowledge was circa, I don't know, twenty nineteen, something like that. So maybe in the past seven years things have dramatically changed, but that's what it felt like. Yeah. Going to to open source. you're you're actively involved in in open source and open source today.

And Yeah, it seems open source is changing a lot. Especially with with with AI and and you know, you're seeing stuff at ghosty. Like can

Can you tell us like how you know open source has has changed with Ghosty, with AI contributions and and what what are you seeing with with open source maintainers? Seems like there's a bit of a you know, like drama or or worrying stuff happening. Well I I would say more broadly, the issue facing open source today, um is I mean there's there's multiple, but the one that I feel is most prevalent across industries right now is AI contributions. And the specifically

the ratio, the signal to noise ratio being incredibly low, or in other words, just being super noisy with low quality contributions. It's just stressing the system quite considerably. And yeah. And and so

After you left left HashiCorp, you you started Ghosty uh how many years ago was that? Was that like two years or so? Well I I left HashiCorp over two years ago, or a little over two years ago. I had like poked around with prototypes at Ghost D like maybe three years ago, but

after I left Hoshakorp I started just like kinda working on it like twenty hours like much more um just because it was the thing that I had. What drew you two go see? What what was your kind of vision and why did you start working on it? It's a it's a it's a better terminal, right? It's a terminal.

Better is objective. Well I I installed it because I I I like it better, but yes, a a terminal, an opinionated terminal, right? Opinionated, um very modern in terms of like supporting as many of the newer specs as possible that enable

functionality like displaying images or um you know clicking on your prompt to move the cursor and but like dozens more uh examples like that. The original thing that drew me to it is is the exact opposite of good advice that people usually give to people, which is that you

Find the problem and you build a solution. And what I did and you pick the best technology that then solved that. What I did was I found a set of technologies and I was like, What can I build with these technologies? I went the opposite direction. And I had spent over ten years, twelve years At Hosh Corp Incorporated.

and three years prior to that doing infrastructure open source. So fifteen years in total, just thinking almost all the time about infrastructure and cloud services and things like that. And so I had felt that I was rusty. I had sort of like my skills have had weakened on desktop software, systems programming to a certain extent,'cause I was so constrained by networking challenges, distributed systems. So like low level systems programming had had had atrophied.

Um, I had never really worked with GPUs and GPUs I guess crypto was happening, but I kind of ignored that whole trend. Um uh but this is pre AI, so Um but GPUs were obviously in use and I I just felt like I had no idea how they worked. So I wanted to go to desktop. So I picked all these like different technologies. And I said okay, Zig'cause it looked cool to me. I just wanted to try it. Kent for those of us I'm I'm not into Zig, I heard good things about it. Can you explain why Zig is

So interesting, innovative, and why does it grab so many and so many devs' attention? I don't know why it grabs other people's attention, but for me it was it it just felt like the the best better C that I saw out there. And I am someone that's coming from the position where I actually enjoyed writing C. So a better C sounds great to me.

To me it's it's not very annoying uh in terms of like, if I want to blow my own foot off, please let me blow my own foot off. You know, a bunch of qualities came together where I thought on the surface it looked cool, but it's very hard to judge a programming language on the surface, so I wanted to build something with it. And so yeah, I I pick zig, GPUs, desktop software. What could I build? Uh for for all my time at Hosh Corp. Um

I built CLIs and I was like, well, I live in a terminal. Like what does it take? I don't I live in a terminal and yet I understand very little about a terminal. So why don't I just like build a toy project that's a terminal? That's how it started. And and as as with a a lot of stuff I find that

Once you dig beneath the layer of taking something for granted, you realize that everything is way more nuanced and complicated than you imagined it to be. And terminals were the same way as once I dug beneath the surface I realized how much they were doing, how brittle some things were. how much better certain things could be and I

I I got sucked into being like I wanna do this better. So Okay, for like someone who's a a dev, you know, like I I use terminals as as well. I'm gonna ask the stupid question, how hard could it be? What does a terminal actually do? And then can you maybe tell us like how Ghost C is is structured or like what what are the things that it it needs to do? Just to give a little empathy of like all the work that you're doing. Yeah, yeah, yeah. I I actually get that a lot. I I get that question a lot.

It's definitely not a dumb question. It's really like it gets asked less now, but a lot of people were like, I thought they were done is usually the most feedback I get. It's like, what is there to do in a terminal? Um so at a basic level, they don't do a lot. The problem is that

the the functionality's grown significantly of what terminal developers want to do. But let me let me just give um what they do. It's kind of like an application development platform, right? It's like a f it's it's not an operating system. You're not dealing with like hardware level problems, but it is like

a application sandbox on top of that and that other applications run within it and need to render text, they need to render colors and images and widgets and mouse events and all this stuff. Like you're

The the best description is it's like it's like a browser, but for text content. And so all of the complexities that a browser has. A terminal has similar ones, a smaller scale, but similar ones. And if you try to extend what a terminal is capable of, then it it gets you know, you start bringing in more and more problems. Like as soon as you brought images into a terminal, you introduce like a whole new ecosystem of problems. But the tongue in cheek answer I like to give to Ghost's complexity is

that it's thirty percent a terminal and seventy percent a font renderer. Um and uh yeah, that's what it feels like. It's really like a problem of uh, you know, that terminal screen you see, whether it's GPU or CPU rendered, that terminal screen you see is like You're drawing on a canvas. So you are building a renderer for tech.

uh in there. Everything kind of bubbles from there. So from a rough architecture standpoint of Ghostie, i I I like breaking it down in terms of threads because if Ghostie is multi-threaded. Not all most terminals are not. Um but I'm not saying that as a positive point, just a good way to describe the architecture. We have a central UI thread, which just draws the windows and stuff. That's pretty standard for desktop software. And then we have an IO thread, which runs the actual shell.

that you're seeing. So any bytes that we send or it sends back to us, it's processed by the I.O. thread. And then we have a renderer thread, which is actually drawing it. So it's it's The best way to think of it is it's on a v sync clock through thirty, sixty, hundred twenty frames per second is it's just sampling what the terminal state is and then drawing it. And the render itself uses a font.

subsystem on the same thread, but we have to take the fact that this grid has this character uh these sets of characters and map them into fonts and do that all on our own. A lot of people think, oh, doesn't the operating system solve that for you? But they don't unless you're much higher level. Like y you know, you can't just draw easily

you know, monospace text in that way. You have to really put pieces together. That's the the big picture. Um it's quite simple at that level and then just, you know, extend all the functionality that terminals have into that. So you're kind of like building a like a two D graphics engine a little bit that has like very focused on fonts.

Yeah, yeah. It's it's a s from a renderer side, it's very simple. The render is actually not that complicated and and I won't o overcomplicate the hardest part is actually maintaining the terminal state. So The way terminals work is they're a they're a grid of monospace cells. So you'll have like eighty by twenty four, eighty columns, twenty four rows, and there's

commands that the program could send to move the cursor or say, if I like to say think of it like a paintbrush. They could say make the paintbrush red and bold, and everything after that is red and bold. and now change it and you're just maintaining the state and drawing around and then there's all the scroll back, right? Which people are used to in terminals going back and That's where the challenge is, is doing that in a fast performant way and that's what I try to do with Ghosty. I mean I

I show this there's so many benchmarks we run, but one of the most obvious ones that shows the speed, which also gets a lot of criticism, um, is just catting a lot reading a large file. If you just like dump a bunch of text, how fast can it get through it? And you'll see a stark difference between modern terminals. I'm not just gonna say ghosty here. Like if if you take ghosty, kitty, um alacrity Um any of these newer terminals, they're all gonna do great compared to

terminal at app on Mac OS um or traditional like Linux terminals. The criticism is Why does that matter? And you know, the the easy answer is when you um when you accidentally cat a file. Like a lot of people will force close. Um the creative Redis posted a great comment for me, a great cred uh comment on Hacker News about why He loves Ghosty, which is that he previously previously used to tail production Redis logs and you know just spews logs out and he used to have to

send them to an intermediary file and then read them out later. So he could render it. So he could render it and actually work with it. And he doesn't have to do that anymore. Because Ghosty's fast enough that he could just let it dump while he's going through it.

parsing it, like like mentally parsing it, things like that. And and that just saves him time. And um yeah. There's something to be said, uh at some point we should probably talk more about the fact that a lot of software these days does not care about performance. And I think it's refreshing to actually have examples and I I hope we will

at some point maybe get back to it, you know, we'll talk about AI but that might not help. But there's a level of craftsmanship, right? Just like not wasting resources or being efficient or I I think we all like I I see in my day to day life like

We have more powerful resources, laptops, phones, and they're not getting any faster and it's just frustrating at times. It's kind of like the love of the game. I mean a lot of a lot of Ghost is just the love of the game. Um like Like I like to say like our renderer'cause'cause like a disclaimer before, like

It's not complicated. I'm not I'm not ever gonna say that Ghostie is like a a two D game because a two D game from a rendering standpoint is much more complicated. Um but I do care a lot about the render and we got our renderer down to for a full screen on on my Mac um set of grids, each frame updates in roughly I don't know, it's like It's something like that. Nine microseconds or something.

Um that doesn't include the draw time. That's just like taking the state and submitting work to the GPU. It's about nine microseconds, and then the GPU takes some time. 120 Hertz, 120 frame per second frame is 8,333 microseconds. So if you have nine, you know, again, we don't have the number of how long the GPU takes, but

it's super perfect it doesn't take much time at all. You're leaving a lot of options and work for what I'm saying is like we could have made it two thousand microseconds and it wouldn't have mattered. It like you would you would still get that performance but

But that's not fun. Like I wanna make it sub ten. I I I I I like it the fun. Yeah. So we spent a lot of time just like it was a big I I blogged about it. It was this thing where we got it down from it used to be about eight hundred microseconds and got it down to like nine and uh I I thought that was awesome, even though for end users it doesn't make a difference. But as you say, the the craft and the love of the game.

So when you started out building Ghost, that that was around the time where I think Chat GPT was out. There were some tools. How did your tool set change in terms of how you're developing day to day? Oh there's two sides to that. So one AI gave a huge boost to terminals, which is a funny thing. Like a f like Oh, how so?

The number because of claud code and all these things, the amount of time spent in a terminal has gone up, which if you told me in twenty twenty three terminal usage would go up, I would say no, it's not gonna go up. Um I I had no disillusions that I was gonna like save terminals. And I didn't, right? Like AI came out and came out with all these CLI tools. And and even when you're seeing like Codex apps and claude apps like

is leaving the terminal, they're still executing so many things in a pseudo terminal. The number of terminals out there has is massively larger than there was in twenty twenty three, which is hilarious. Oh wow. Yeah.

So random. Super random. And so that's part of why uh one of the things I'm doing with Ghostie is extracting. It's actually extracted already, what I've called libghosty, which is everyone reinvents this very small surface area of a terminal and because they do it, it breaks like all sorts of things break. Like if you run a Docker build or push to

uh platform like Heroku and you do enough weird things in the terminal that aren't actually that weird, just like draw a progress bar, it renders it like chaos. All over the place. All over the place. Yeah. Um And it's just because they've poorly implemented a tiny subset of a terminal because they're more complicated than people think.

And so LibGo C is this minimal zero dependency library that people can embed terminals anywhere. Oh, cool. And yeah, yeah, MIT license and just it's really like I'm tired of seeing broken terminals everywhere, so please use this. Um so okay, that's the one angle. Really funny. But the other angle is actually A AI usage. It's hard to say I'm a I'm a big fan, but you know within the right categories of things. Like I think that it's a revolutionary tool.

and I get a lot of joy using it. Yeah, I use it every day. Um I use tools like Cloud Code and AMP and Codex and and the chat tools like every day for some aspect of my life and it's really allowed me to choose what I want to actually think about. Right. I think that's the most important thing is that I always felt limited in terms of, oh, I'm gonna have to spend the next two hours I don't know, doing this boilerplate annoying stuff and that I don't want to learn about.

But now I don't have to learn about it, which is yeah, I'm not I'm not like getting skill formation in that category, but I could now spend those two hours Doing something else.

the best to me. In your workflow, do you just use a single agent? Do you use multiple agents? Have have you have you experimented with them? I've tried a bit of everything. I would say my standard workflow, what I try to do is I try I endeavor to always have an agent doing something at all times. Uh maybe not when I sleep. I don't go that far. A lot of people do go that far. I don't go that far. Um but while I'm working, I basically say

I want an agent if I'm coding, I want an agent planning. If I if if they're coding, I want to be reviewing or uh you know uh that there should always be an agent doing something. So you have it in a sess in a separate tab? Yeah. separate tab and and sometimes it's multiple. I don't there's a lot of work that I do around cleaning up what agents do and I don't run like gas town esque

like things and so I'm the the mayor, so to speak. And so I don't wanna run too many. I don't find it that fun to clean their stuff up. But periodically I'll I'll run two um in competition with each other because I it's a it's a harder task and I I don't have a high confidence that they're gonna just like crush it. So I'll just run Claude versus Codex or something like that. Or I'll have one coding

I'll have one doing like some sort of research task. Um I absolutely love them for research. That's awesome. Um and then I'll be doing something else. No more than two, I would say. Yeah. The code that they generate, do you always review it or?

Have you kind of got a bit more loose and you know, some people swear on having a uh closing the loop, having validation for it, or are you like still like, all right, I'm I I wanna see the exact code and I'm I'll review if it's correct and what I expected? Uh matters what I'm working on and

If it's ghosty, I'm reviewing everything that's going into it. Um, if it's like I set up a personal wedding website from one of my family members, I don't care at all what the code looks like. Did it render right in the three browsers that I tried? Yes. Did it render right like on my phone? Yes. Don't care what the color. Doesn't make any net requests, no, has no secrets access. I don't care. Like ship it.

That's only gonna be online for two months, so ship it. Yeah, and then how did the AI policy at Ghostly change? I remember that in may maybe a year ago or so, you asked for disclosures if someone is using it. And just very recently you kind of crammed cracked down and said like, All right, no more. Yeah, we're gonna change again to Well I'm not gonna change, we're gonna iterate. Um so yeah, a year ago started asking for disclosure and people you know, th the the very fair

question there is what does it matter how the code is produced? And the reason to me it always mattered was because it dictates how much effort I go into fixing it. if you produce the code of the AI and you did it really quickly then I'm not gonna spend hours Fixing up your code. You you

you spend your time. Yeah, uh'cause'cause you know that that that person didn't put so much time in it, not much human time. You're kinda trying to mirror it right or ever for effort. If you put in hours, I'm gonna put in hours back and I'm gonna help you.

But if you put in a few minutes and never read anything and threw it over the wall, then I should be able to read it in a few minutes, say no thank you, and close it. It's it's fair and I need to better understand what that is. And, you know, it's not about bad code because open source has always gotten bad code contributions.

But the difference before is usually those bad code contributions came from people that were genuinely trying their best and put in a lot of effort just to get to that bad code point. And so I I uh people behave differently. I would always try to reciprocate by being like, This is someone very junior or this is someone just new to the project and I would try to educate them, be like, Okay, we should do this better and and give these careful reviews.

But if it's bad code that there was low effort, I'm not gonna give a careful review. So again, like I wanted to know these things and and the disclosure worked decently well. The issue wasn't the disclosure, the issue was that the quantity of low quality AI PRs that we were getting reached.

a a a a point where it was too high. Like do you know why that might have happened? More people instructed agents to contribute a PR to fix an issue they had? Like, do you have theories or actually like like seen evidence of why this happened? I have theories and I've seen some evidence, but yeah, I I mean I think obviously there's the rise of just AI usage in general, but the real trend that

a step change that I saw at a certain point and I don't know when it happened because I don't use agents in this way, but at a certain point they started opening PRs. You know, before it was like you generate code. And maybe they commit and stuff, but you would still like push it to a branch and then open the pull request. At a certain point they started opening PRs.

And there is a dead giveaway at AI because at least to this day, to at the point we're recording this, the way Claude opens a PR is it opens a draft with no body and then it edits a body later and then reopens it for review. Which is not how a human would do it. Oh, like one human a year would do that. And now it's happening three times a day.

And so even if they're not disclosing AI or they're hiding it, it's like oh, and it happens at a speed that's unrealistic. It opened, the body came in less than a minute later and it opened less than a minute later. Like Pure AI. I I just tweeted about this a couple of days ago, which is just like I I wish that these agentic

tools would put a pause on opening PRs for a second. Um, because I think that's the point where it's really causing a lot of friction. How did you change the policy? Are are you Considering closing down PRs, you mentioned that recently, that you've you you've the the the thought crossed your mind. I would say I was crashing out in that moment.

Uh but I uh but kind of. Um so we shipped this policy update where PRs written by AI are no longer allowed anymore unless they're associated with an accepted feature request. So you can't just drive by and be like, I did this thing that I've never talked to you about. Here you go. It we and we we get about two or three of those a day. And so we just close those in. I don't even I d literally don't even read the content. I could see it's AI.

Uh I can see there's no fixes issue number. I just close it. No idea if the code is good. Don't care. It's just policy. Don't have time for that. That's pretty much where we landed on currently. And uh we're recording this in the middle of another transition, which I already have the PR open, um, where we're gonna switch to a explicit vouching system for the community. So you're no longer able to open a PR at all, AI or not, don't care anymore, which is I think the people who

criticize where it came from doesn't matter. It doesn't matter anymore. Now all that matters is that another community member has vouched for you. Um and if they vouched for you, you're added to a list where forever or indefinitely you could open a PR. If you behave badly, then you, the person who invited you and the entire tree of people they ever invited

are blocked forever for the repo. This reminds you a little bit of do you know the social lobsters. Lobsters. Yes. That's what it's based off of. So the idea is that you're putting your own reputation on the line by vouching for somebody else. I'm a reasonable person. If if this happens then I or one of m our maintainers or community made a mistake, if you just like hop in a Discord or email and and

seem like a reasonable, apologetic person. Like I I'm not gonna spend a lot of time like there's not gonna be like a I don't know, a mock like court type session. I'm just gonna be like, okay, I'll give you another chance. So yeah, we're we're sort of moving to the ass system. I think one thing that's a little bit different is um I should say that this is one inspired by lobsters, but specifically in the AI space is inspired by this project called Pi.

Um they do this. Oh well they do they call call this mill on pie. It's a self approving sell like uh build your own agent toolkit. Um so, you know, kind of ironically it's it's an AI tool, but they care a lot about code quality and anti slop and things like that. So they have a similar mechanism, a little bit less of the tree and some other but a similar

You can't open a PR unless you're vouched for. And the other difference here that we're going with is in addition to vouching where you could positively mark someone, you could actually denounce users. So if there's a bad actor, you could actually ban them, not not just like you can't even attempt to contribute again. And um that's just a yeah. We had one yesterday where so on Open PR, we closed it because it violated it they had no associated issue. Uh and it was A

And then they just reopened it. Like not the same one. They resubmitted a new branch and reopened it like less than ten minutes later. And I was like, oh my gosh. So um stuff like that is just it's the problem is it's just wasting time. It feels like most of open source will have to change because of AI, right? Like it's

Yeah, y you you probably know about more more more maintainers, but I I hear this is your story is not the only one you know like the project closed down uh PRs. GitHub is I think just shipping a feature that projects can re automatically close or reject PRs. Yeah, I think open source will have to change in a lot of ways. I mean, I think I f I forgot who wrote this, but you know, one of the logical extremes is if agents are so good.

You don't need open source anymore'cause you could just build it. Right theoretically, yes. That's a that's the extreme. I don't ascribe that extreme, but that's one of the extremes. The issue is there used to just be this natural back pressure in terms of effort required to submit a change, and that was enough. And now that that has been eliminated by AI.

The I like the wording that Pi uses, which is that AI makes it trivial to create plausible looking but incorrect and low quality contributions.

And that's the that's the fundamental issue. You know, uh open source to a certain extent has always been a system of reputation, right? Like you you earn some trust and you get more access that you know, and uh that's how it's supposed to work. Um, but yeah, it's been that reputation system has been taken advantage of in a certain sense with with AI. Um or the default allow PRs has, you know, has been taken advantage of. And so I think

Like like this vouching system that that we're proposing for my project. I think it's like very true to what open source is, which is that open source has always been a system of trust. Before we've had a default trust and now it's just a default deny and you must get trust by somebody. Do you think we might see a lot more forking happening though? I hope so.

I hope that because until now forking used to be a you know, like a like a fork off a little bit because it was a lot of effort, uh it wasn't to to keep up. Like it it never seemed viable to fork a profit or project, right? Yeah, and I and okay, I I am separate from AI and everything. I

I've always been a huge proponent uh or I guess in the past few years I've been a huge public proponent of m there should be a lot more forks. Like a lot more forks because open source I think One of the reasons maintainers have been taken advantage of to some extent is that contributors have some sort of entitlement, you know, whether it's toxic entitlement or not, but there's some sort of entitlement which is I've made a valuable change.

So you should accept and it's clean and it works great, so you should accept it. But You really don't have to. Like you you absolutely don't have to. And then I've seen this time and time again where you have a high quality PR, like perfect PR, but you say no. And there's anger in the community. But the thing is I I've said this since

Ten years ago in the Hoshcore days, hitting the merge button is the easiest step. Getting getting to and hitting the merge button is the easiest step. Like undergraduates should be able to do that. It's after that, it's the years of maintaining whatever you just merged within the context of your

your roadmap, um, the bugs, um, customer needs, all that stuff. Like, that's the hard part. Like you're signing up to keeping this forever. It's very hard to remove features. So or anything, remove anything. So the core privilege you get with open source, like OSI open source is four. And you should take the that's that's the right you got. You should fork it and maintain your own software. Yeah. One interesting impact of AI, as someone tweeted about how

There's a rumor that big tech is looking into re architecting their monorepos because of agentic tooling uh AI tooling, just a lot more code being turned out. What's actually what's actually happening? What what's the problem with Git? The problem with Git I mean there I I think there's a lot of problems with Git, but uh the monorepo problem with Git is that

Git is is relatively bad at very large repositories because you you pretty much have to clone the entire repository. There's there's some extensions that like fix that, but like official mainline Git can't really do that, right? And so for very large uh changes, the uh very large repositories

Um it's sort of annoying to maintain. And then if you have a lot of churn in it, it's very hard to get changes into whatever your trunk is, your main, your master branch, right? You concept to rebase, merge queue solves that to a certain extent. I think merge keys works for humans at a certain scale, but the merge keys could get quite deep. But then if you sort of 10x that

like conservatively, I think ten X that. And then if you buy into like hype cycles and you hundred or a thousand X that, I think it gets completely untenable in terms of how are you ever getting any semblance of cohesiveness onto the main branch um quickly. And and so Yeah, I I think there's a confluence of problems there, which is which is the merge queue problem.

the disk space problem, the like branching review type problem. Oh, I I also treated the other uh another time where Like Git has this. you you branch and you push up your branches, but the branches are only the positive. Like when you when you close a PR and you you don't accept it, like

You pretty much are the branch. In GitHub, you could re-access closed PRs, but you a lot of people don't even get to the PR stage. They experiment, they're like, oh, this isn't the right way. And they never push the branch.

And and that's like relatively important information. Relatively important. I it's not as important as the positive, but there like I I think there should be a lot more branches and get a lot more information that we just never throw away. Like we're at to me We're sort of at the like Gmail moment for email for version control where like you used to really have to like curate, delete all this email and then Gmail came out.

gave a gig away for free to everybody. Never delete it. And that's where I feel like we should be at with code, which is like just this huge repos, a lot of context. We need better tooling in order to find relevant context in that Git repo, um or version controlled repo. I would say that the real you ask for like real examples. I do advise um a

company that's currently stealth but working in this space. And there's the real examples is is is driven by the highly advanced companies. The companies that are like going really all in and drinking the Kool-Aid and they're struggling in terms of the amount of churn that these agents is causing is so much greater than humans. And it's not a AI review problem or anything. It's really just like a release problem. Like Managing the merge queues.

humans getting access to the right set of data in the repository and things like that. So are are problems performance problems uh mainly with w with with Git or or just like even the workflow of Yeah, yeah. All of it. Performance for sure, but workflow, yeah. I mean like every time you pull your

You can't y you can't push because every time you pull there's another chain. Like every time you push it's a big thing. Do you think Git will be around with with with these Jet Jackets in a few years? Who knows but What's interesting is this is the first time in like twelve to fifteen years that anyone is even asking that question without laughing. We're not laughing. Right. Like like if you if five years ago you said we'll get be around in five years, you'd be like, Are you

Yeah, of course it'll be around. Like that's crazy to think it right. But now people could ask that question. And of course some people will laugh, but like there are people that critically think that Git might not be around in five years. Well, I think you do want to save the prompt history because often reading the prompt is actually if it's a bunch of code generated, the pull request is meaningless. Changes will ha like Git and GitHub. Uh forges in their current form do not work.

with a genetic infrastructure today. And it's nascent today. So Yeah, where it's change will happen. And I'm not exactly sure and that's not something I'm trying to change myself, but it it you know, I'm on the receiving end in terms of a agent user and a maintainer where I'm like, this isn't

working. What other engineering practices that you know we have been relatively stable for like ten, twenty or even more years you think have to change or or are looking to change, thinking things like C I C D testing, code review, other, you know, ways of Yeah, uh AMP has the saying which is is is

it's kinda clickbaity, but it's so sure as everything is changing and this is this is the first time really where it feels like it is the first time in my you know Short, relatively short to other people, but still a twenty year professional career. that so much is on the table for change at one time. And I'm an optimist, so it's really exciting to me. Um I I it's a lot of fun, but

It's we've never seen so much editor mobility. Editors used to be one of those things that once someone picks an editor, it's very hard to get them off that editor. They're like stuck. The level of editor mobility in the past few years between like VS Code and Cursor and and just jumping around is Is unreal. So there's a bunch of mobility there in terms of uh yeah, I mean cursor itself is a great example of a company that

Reach an insane valuation that you could never have gotten pre AI on an editor product. So editor forges, um, CI C D for sure. And I I think that testing in general, because to make an agent better, it needs to be able to validate its work. And so tests go from even the best test case scenarios don't have like

I mean the best I guess have full coverage, but th th that's a very extreme. The the very good test case scenarios, just test like one of the edge cases and one of the happy cases and, you know, bad case and th they they just kinda go through and and if it passes it's probably good paired with a human who's thought about the problem. But AI is more goal oriented in terms of I want this feature to work this way, that if it doesn't see a spec somewhere or a test somewhere that

other things should work in a different way, it'll just break it on on its path to its own goal. And so uh I've heard this called a lot of things. I mean, the one I like the most is like kind of like harness engineering. Which is like harness engineering. Yeah, it's like I and I've been one of my like goals for this calendar year has been to spend more time doing that, which is that anytime you see AI do a bad thing

try to build tooling that it could have called out to to have prevented that bad thing or of course corrected that bad thing. And so it's sort of like moving from the product to m working on the harness for the product or product development. And so Yeah, there there's there's a lot of that where I think testing has to change to be far more expansive, but CI C D is not

set up just resource performance wise to be able to do stuff like that. Um so yeah, I'm I'm not sure how it changes, but that's gonna change too. So everything is on the table. It's really interesting. Yeah. And it's a lot a lot of tools to be built. One other thing

Uh observability. Yeah, and then and I guess on that same topic, I mean, of the volume and and scale and observability, it's also like the sandbox like I didn't think even being in infrastructure and being heavily into infrastructure, you know, containers blew up the amount of like minimal compute units we had, like floating around everywhere. I didn't think that was gonna go up. I mean it'd go up like predictably up, but I didn't think it was gonna like slope change up.

And it has like sloped change up already just due to the sandbox environments that agents need. And yeah, I mean that's super interesting to me'cause that stresses a whole lot of new systems. I I think, you know, the things that I worked on, like all the products I worked on, but also

things in the ecosystem like Docker, but like Kubernetes, they're gonna be stressed significantly because they're engineered for some level of scale, but this is a different type of particularly non production workload scale that you have to support. So um yeah, it's it's Fun, fun problems. Going back to hiring, you you've hired a lot of engineers and

You previously talked about something really interesting. This was I think in the context of maybe HashiCorp, how some of the best engineers you've hired had really boring backgrounds. C can you talk about that? Like who who were the best engineers you hired and like how how's a better way to frame it. Yeah, I I I I stand by this. Mo the best engineers I can remember

from my time at Hoshcrap but also just in every job that I've had are notoriously private. And not because they want to be private, because they just don't care to be public, I guess would be the better way to put it. I don't want to like

carefully describe anyone without giving them away. But, you know, they're just they don't have social media profiles. Uh, very often they honestly are nine to five engineers. They go back and they don't code at night. They just spend time with their family. But because they don't do anything else during their working time, they're like locked in and

And they're really good. And it's not about putting the hours. It's also just skill wise, um, super strong. Um, so yeah, I always found like When I when I was reviewing resumes and stuff, when you find the person that has a resume where they like

they don't have any GitHub. Even a GitHub account. Like some people are like, Oh, you have to have public contributions to to stand out. Like that is a way to stand out. But also if you have zero public contributions and you've just worked at companies

that also I've never heard of before. It kind of is interesting to me, which is like, okay, you you might know something like deep. Um, so Yeah, I I I think that you know, the problem is and I the funny the ironic thing is I spent a lot of time on social media.

And these engineers are better than me. Um but the the funny thing is every moment you spend on social media, time is zero sum. So any m every moment you spend on social media is taking away from something else. And and the issue is it's not one for one because as every engineer knows, the time it takes to really get your mind into flow to get going with something is

It varies but it takes time. And so when you context switch to social media, if you if something's compiling and you tab over and you spend time, you you've given something up in terms of thinking. I I think one of the best things I do spend a lot of time on social media, but maybe unhealthy um ti amount of time on social media, but also an unhealthy amount of time um at night.

I I don't have insomnia, but it takes me a long time to fall asleep and And it's because I just sit there in the dark and I love some people do this in the shower, but it's not long enough for me. I love to just sit in bed, the lights off, my wife's sleeping, and I just think through like I'm writing code in my head, I'm thinking through products, I'm thinking through website copy, I'm thinking through I'm running CLIs in my head of how it's gonna feel.

And sometimes uh last night I I went to bed at nine thirty'cause I've I'm a I'm a dad, so I go to bed early. And you don't know when you have to wake up. Yeah, yeah. And I didn't even feel like I was up that long.

And it's like, oh I gotta go to the bathroom, I should go I should really actually like go to sleep. And I looked and it was twelve thirty. And all I was thinking about was uh it's so dumb. But all I was thinking about was this vouching system of how vouching might work, it might not work. And I've always had this thing where I'm willing to I like competing. I I think competition's fun. Um but

I always feel fair game to compete with anyone in product building space because I think I'll spend more time thinking about it than they will. I think people turn it off and I I don't I try not to turn it off. So um yeah. I mean I I think the point of all that is The best engineers are the ones that Context switch the least probably. Having used AI, AI agents, you think that's my change?

Because you know, like these agents can go on and think or or or do work for you. Like how would you hire in this in in this new world where we're using AI is kind of a given. Most devs will prompt uh and fewer and fewer write, even though best devs clearly know how to write code as well. Um, I would definitely require competency with AI tools. You don't need to use them for everything. Uh uh that's not important to me, but

It's an important tool to understand the edges of. Like it it's like any other tool where sometimes it's useful and sometimes not useful. But if you ignore it completely, you're gonna do something suboptimal in a time. I mean, the best example to me is pro proof of concepts. Like constantly in real product organizations, you have an idea and you need to like demo it out to figure out if it works.

I would much rather someone just like throw slop at a wall that you're never gonna ship and spend a day doing that, uh you know, maybe less than a day doing that, rather than spend a week doing it or organically as as a human, like'cause you're gonna throw it away anyway. And you don't even you might throw it away because it's a bad idea, but I'd rather prove it out. And so just slop it up. And so this is why it's so nuanced. I'm so like I'm so

get so worked up about sloppy PRs to open source. But it's because there's a time and place for them and that's not the time and place for them, but there is. And so I would hire in that way. And I think the other thing that I don't know if it's the right thing to do, but I would strive that that goal that I have, I would strive for everyone to have an agent running at all time. Again, like it doesn't need to be coding, but to be doing something extra for you. I would strive for that because

Uh I I do a driving. That's my biggest one. On the drive here I had some deep research going and it's like I will always spend thirty minutes on the boundaries when I wake up and before I stop working or before I leave the house or something, I spend thirty minutes stop working. What can my agent be doing next? That's that's slow. What's a slow thing my agent could do for the next

time and I knew I was gonna drive here for an hour. It finished far faster than an hour, but you know, it was just like, Oh, I need to do some Library research. Um, okay, find all the libraries that have these properties that are licensed in this way. And I was looking up some like H D B three stuff, quick stuff and so fine build that ecosystem graph for me. Um, right before I left I was working on

Something to do with this vouching system and I didn't quite understand the edge cases of what I was doing. And I will think about that manually, but why not just start just start an agent to like

look at this repo. And I use AMP to like consult the Oracle, like think deeply about um what the edge cases might be. What am I missing? If I had another two hours of work, I wouldn't need the agent to do that. I would have done it myself. But I don't, so why not have it do it? So It's just part of my goal to always have one going. And I unfortunately don't have one going'cause they finished it all right now. But

In in interesting. And so this agent running there is kind of does do I feel correctly that it's now so natural that it doesn't get in the way of your own thinking? Like you do your own thinking and you do your work, but every now and then you glance and you you ping it or you start it or It's it's now it's so it's not distracting, right? Because I think that's Yes, I actually turn off all the agenda tools do this, and I turn off the desktop notifications.

Yeah. Um I I think the dust on notifications are for the most part a mistake. Um so yeah, I turn those off. I choose when I interrupt the agent. Not it doesn't get to interrupt me. Um so for sure. And and then there's another aspect where I think my engineering has changed where I try to identify the tasks that don't require thinking and the tasks that do require thinking and and just delegate, like delegate the work to an agent. Like it sometimes

it just feels productive to do the the non thinking tasks and you're like, Yeah, I did a lot today, you know, got this but but a lot of times I just try to just delegate that out. There's a lot of people that that, you know, say like you think less and I think if you use the tools wrong, you do think less because you just like launch an agent and I don't know, go watch YouTube or scroll social media or something but

if you instead view it as a way to choose what you think about, then I think that you don't need to sacrifice that thinking. But I think the the problem is the majority of the population probably won't do that. Yeah, but it's still uh I think it's good food for thought. And

It's good to hear from you on how you're using it. It's working for you. W w w when did you start to to have this second agent running? Welcome to Switch. Was it the models getting better or Yeah, i i I don't remember which model it was, but there was a certain

I tried Cloud Code right when it came out. It was just like March or May last year. Yeah, it was March, the beta, yeah. And the May public release. Okay. I I don't think I used the beta, so it was probably May. Um wasn't a huge wasn't super impressed, honestly. Um And then I mean really quickly by like the summer, at some point during the summer oh, I remember. I remember. Um, I saw so many positive remarks about it that then I started to get scared that

I would be behind on how to use a tool. And so I actually started forcing myself to I I still didn't believe in it. So I would do everything manually, but I was forcing myself to figure out how to prompt the agent to produce the same quality result. I was working much slower because I was doubling the work and it was more than double because

It was they're slow and it's we're going back and forth and I already had the work done and all this stuff. But I was forcing myself to do it. And you find stuff that I couldn't figure it out. I couldn't like it just wasn't there yet. But then I found other stuff where it's like, oh, I naturally got to the same point that thousands of other people got to, which is like, oh, if I do a separate planning step

It does so much better. And everyone got there. And then I figured out, oh, if I have a better test harness for it to execute. it does a lot better. And then, you know, I I think everyone starts with like no agents.md or claude.md or anything. S same thing I realize, oh, if it makes a mistake and I add that just to agents.md, it never makes that mistake again. Like, oh, and like these these are just like incremental things.

that I recognize when I see people that are new or I've watched a couple of live streams, like lurked on live streams where where like kinda anti AI people like try AI and it's one of those things where I'm like they're just swinging the hammer way way off, right? Like it's because you haven't it's the the thing is like it's six is it's as if someone tried to like adopt git.

and they used it for an hour and decided they weren't more productive with it. Like it takes much longer than an hour to get proficient with Git, but you put in the effort and then you reap the rewards later. And it's sort of the same thing to me with AI tools. What what would your first advice be for someone who's like not My first advice would be reproducing your work with an agent. And if you really, really don't want an agent to code

reproduce the research party work with an agent. Um, like there there's a lot of people it's like, I don't want it to write code for me for whatever reasons. Like, um, but yeah, just kind of delegate some of the other research part. There's so many places it could be helpful. So it it doesn't need to take you know, you don't need to pick up on the it must replace you as a person kind of propaganda.

you could just find the the the corners of where you work and and replace those parts. One thing that you you give people is you give advice on for Potential founders because you're a successful founder. You you've had an exit. You built up this awesome company. You get a bunch of emails from people asking, Hey, I want to be a founder. What was your advice?

And you you wrote about this, you you shared the email, but but can you tell us like what advice you typically give people and how is it received? Uh well I usually ask for something more specific. Uh because yeah, if someone's like, What could I do to be successful? One I'll always disclaim that you're consulting someone with survivorship bias. So you need to take that into account. Um but I'm willing to share my experience as a survivor, but just understand that there's survivorship bias.

Um but usually I ask for like what's what's something more specific? Like what are you trying to do? Um and so we usually get to like should I open source my project or not? Or should I be remote or not? Or should I do enterprise and and I don't know. Um but my

My the the most general advice I usually give people is Startups are much longer than you think. Um, you're gonna probably work on it for I I say imagine ten years. A lot of people say five years, but I say imagine ten years. Like, is this really something you wanna work on for ten years? And is it something that like You need to have a certain amount of hubris in order to say, I'm gonna work on this for ten years and I

Truly believe I'm gonna do it better than anyone else. There's nothing behind that. No substance behind that other than tubrus. So you need to have a certain amount of of ego and hubris in your head to make that, but not too much where you'll be blind to change coming in. So that's usually like the first advice I give'cause a lot of people have cool ideas, but they're gonna burn out, you know, relatively quickly. So

Uh that's where I start. So currently you're advising some companies. What are you seeing with them? Like m what are what are stars doing these days? What are they doing differently than, you know, like earlier? How's that landscape? Uh again, uh it's really contextual in terms of like if you're an AI startup, it's it's very, very different. Working differently.

there's a lot of pressure to go faster than I've ever seen any startup go. Um I I think the industry is moving so fast that I I I don't advise any AI startups but I've talked to some of them and it it it's even as an advisor I feel like it's too much pressure because

they are just being pushed to prove themselves quickly. Whether it's through traction or revenue or something, it's sort of like there's this mentality within that ecosystem where AI should allow you to go crazy fast. And in addition to that

there are a lot of companies moving crazy fast. So, um, the change is happening. I think that's the one thing. Outside of that, I mean, like I said, it's it's just it's just a ton of opportunity in every space. Otherwise it's a lot of the same stuff. I mean it's remote versus

Non remote open source vers not open source. Do you see the role of software engineers changing now, especially at the AI native companies where engineers like like yourself, they're actually being way more productive. They they can produce a lot more code, a lot more

output are they being pushed into being like, you know, like wearing more hats, talking to the business, of being a bit more like a mini founder, if you will? I hesitate to say more productive. I I I view that there's an expectation that they could do more. I don't think that's necessarily more productive, but it's more like

you should be able to, for example, build a full demo design everything for your pro you don't need a team to do that anymore, right? Like you should be able to do that, at least from a demo perspective. There's no reason not to because again you could ship slot for that. That's fine. I mean this is still the same, but you should be able to research effectively and and in a sense to handle more vague tasks. Uh I'm seeing that a lot more, which is like just

the capacity to experiment is so much higher, I would say. But then when it turns into productionizing something, uh, it feels similar to what it's always been. I I think that there's a lot of companies that are Eating the yeah, the dog food of of Of the AI companies of shipping

Whatever. And I think that's a little scary. Yeah, they look at Entrophic and they're like, Oh, they build cloth co work in ten days and it'll be billion dollar company and they're freaking out of why they're not doing that. There I I think a big change is from like a pre seat perspective or s yeah, pre seat perspective where you would be like, I need to raise a seed in order to build the prototype. That's like

like show me the prototype because yeah, you should be able to build that really quickly. For most things. There's still hard tech out there that you can't do that. So you do a bunch of coding, you do a bunch of thinking about coding as well, ev even as you're trying to fall asleep. What refills your bucket uh outside outside of coding, outside of tech? Obviously like the the the stereotypical things like just

taking breaks and being with my family and things like that. But I mean I think the biggest thing is, you know, I am introverted, so just quiet solo time um refills the most energy for me. I live pretty close to the beach and just if I'm in a bad mentality, things aren't working, I'm feeling unproductive or somet something's going on, like just closing my laptop and taking a walk outside.

that like stuff like that helps a lot. I have a lot of hobbies and stuff, but it's it I think like just as a general recharge it's it's that more than anything. I know there's a lot of people it's like going out with friends or something like that, then I like that, but that's not the full recharge for me. And what's a book that would you recommend, and why?

Um, yeah. I d I pretty much only read fiction outside of news. Um great. Great. Okay. Uh the most recent book of fiction I read is an older book and it it is an easy read. So I hope people are like not like, Oh, he he's an idiot for reading this. But um it was uh what is it called? The

The something life of Addie LaRue. It's just like kind of a romantic type of fiction novel, but yeah, it's just about it it's about I think it's like ten years old. It's older now. Um but uh it's just about a a woman who kind of sells her soul to live forever, but the cost was no one remembers her once they walk out the room. And yeah, it's just going through her whole life of

losing all human connection, but she gets to live forever. Um, what that is like and I know. I I like reading fiction, so I I like reading fiction at night. It I I don't know. I don't know if it's escapism.

Or just like you just like, you know, you gets a little bit different. Well it's still so different to the coding or anything. It may maybe just help it turn off the thing. I I personally I probably read way more f f fiction than I do professional nonfiction, honestly.

Yeah, yeah, I'm I'm the same way. It's my version of T V too. Uh uh uh T V to me is more of a social activity. Like if if my wife wants to watch something together, like we'll watch a show. But if I'm alone, I'm not gonna watch a show. I'm gonna r read probably. Awesome. Well well thanks so much for

Going through all all of these details. It was just not great to hear from how you're working, the history of of Hashi Corp. This was all just reasonably interesting and motivating. Yeah, thank you. Thank you. Hope you enjoyed this long and interesting. Michelle. One thing that really stuck with me from this conversation is Michelle's own rule for himself.

Always have an agent that does something. Not necessarily coding, just doing something. For example, while he was driving to this podcast recording, he had deep research running. Before he leaves the house, he asks himself, What's a slow task that my agent could do while I'm gone? An important part to all of this.

he turns off all notifications. The agent does not get to interrupt him. He interrupts the agent when he's ready. Michelle is in charge and he has a buddy who does the work that he has delegated. While he focuses on the problem that he is solving. This is a nice challenge for anyone listening. Next time you step away from your desk, before you close the laptop, ask yourself, what slow tasks could an agent be doing?

while you're gone. If you enjoyed this episode, share with a colleague who's thinking about where software insurance could be heading. And if you've not subscribed yet, now's a good time. We have more conversations like this one coming. Thanks and see you in the next video.