Navigating DevOps Challenges with Cory O'Daniel - DevOps 191

Hey, what's going on everybody? This is another episode of Adventures and DevOps and I'm your host for today. Will Button joining me today, I have Corey O'Daniel, CEO and co founder of mass Driver, but also from Let's See. You're a principal software architect at the Real Real Cloud Solution, architects at Container Heroes prior to that, staff engineer at Click Trips, and most importantly, taco afficionado. So two things jumped on my mind right away.

Number one, I need to know what your definition of the perfect taco is. But also I want to talk about your background because it looks very similar to mine in terms of the amount of time you spend at each company, and I think that's an important thing to share with everyone, especially people who are just starting their careers. But we'll get to that meanwhile, Corey, thank you for joining me on the show today. Yeah, thanks for having

me and opening with a taco question. Oh okay. I think my favorite taco, like my go to is just like a classic like Carne Sato Street taco like that is my happy place and I've found the place in La that's like one of the better ones I've had in my life. So that's that's very convenient. It's a place called Cacau, Mexico, Testa, and the thing is how you pronounced the second word in Eagle Rock. It's amazing,

and their molat is delicious right on. I would I would be I would be really really sad if you told me that you were living in LA and couldn't find a good taco spot. I mean the thing that it's like I always feel I mean, of course La is to have a can taco spot, right, But like I've I lived in Central America, Mexico under okay, the place were all So I feel bad saying that. But I mean

like there's like this like interesting thing that happens in La. There's a whole bunch of people here, like like California is like what thirty plus percent Hispanic, right, and so like there's a whole bunch of people here, and then all the rich people are bougie and expect the best, so like all food has to be fantastic, right, So like I feel like those two things combined for like just it's like like La is like the best food mall,

like everything's delicious. So I say that and I always feel bad. So I'm like, I've had some really great tacos in Mexico City, but like one of the convenience of driving tenants definitely impacts it. Oh for sure,

absolutely, yeah. Yeah. Cool. So let's talk about your your job history real quick, and I'm going to be real clear, I'm not trying to call you out on it, because, like I mentioned mine, it looks very similar where a lot of your your positions you were there for you know, a year and a half, two years, three years, and I, you know, like I'm older and like my parents were, like, you get a job and you work there for forty years and your

golden and I've never held a job for more than three years, and I think that's just I think part of that is just the new norm because after three years, Like the way I approach it is that if after three years of working with you, I haven't accomplished what you hired me to do, then I have failed. What's your take on that. I've got a couple of takes on this, but a lot of hot takes, So apologies and advance to anybody, but uh, I think one we have a lot If

you think about our job is software engineers. Can you just classified kind of all of us under the software umbrella, whether you don't know. So some people don't feel like software engineers because they're like DevOps engineers and themal engineers, but you are, like there's a different abstraction layer, different level, like

different expertise. Like if you think about like us in the grand scheme of things, we're an interesting group of people where our job is being a really good tool for the most part, right, Like if you think about where software started, where it was people that were professionals and they use software to do their job. Right, slowly software development became its own profession, right, So I see what we do a lot of time is like like like

what's the best way of phrasing it? Like we are brought in to help business people code five things, right, Like that's really what we're doing.

There's people that have an idea of how to make money and they don't know how to do the work that we do, right, and so we're not always invested in the business idea necessarily that we're writing software for, right, And like you can see this is plenty of companies, Like I know one of the companies I work on very early it's not on my resumes because I was a consultant at the time was honest company for sending baby diapers to your

house. There was a bunch of single men there that built the software. None of them were like, wow, I can't wait to get some diapers

doing that, Like that's the first thing. Is like like you can love what you do but not necessarily love the thing that you're building, right, and that has interesting trade offs, like you might get to the point where it's like, okay, like all the ideation and collaboration with the business book, which is kind of like what I really like about software is like talking through the ideas and figuring out like how to make that code, not necessarily

typing it. You to a point where it's like, Okay, we've done the ideation, We've done the interesting things. Like I feel like I'm really in a maintenance mode. I don't love the business, so it's like, okay, well what is here for me? Right? And a lot of a lot of careers we see people working ten, twelve, thirteen, fourteen years, they don't have that choice. I'm a lawyer. If I don't like being a lawyer anymore, I'm kind of shit out of lot. Right,

and so I think that's one part of it. But we're also we're all hobbyists, right, Like we all it's oh, it's five o'clock, it's time to get off work. What am I going to do? I'm going to continue to sit at my computer and write some open source software. I'm going to go play with something, right, Like we love to play

and learn new things. I think like those two things, especially for me, like I have a almost like a wonderlust, like I have to be doing something more, doing something interesting, engaging myself when I get to a place where I'm like, okay, like there's just not really anything here that rewards me. And guess what, the businesses aren't either. The best way If there's any juniors listening to this, like, sorry, guys, here's here's the news. The best way to get a raise is to quit your

job and go someplace else. Absolutely a three percent raise, which means you're losing like eight percent of your salary over the last couple of years with inflation.

The best way to jump twenty percent is get a new job. And at the end of the day, like I like writing software, but you know what, I really like getting paid well, right, and so it's like if I got to the point where it's like there's something that I can say I did, there's somebody at that company that I can use as a reference to, like Corey Rocked, and it's like, Okay, I've gotten like some like interesting raises and maybe I'm like in line with inflation, but

I'm not I'm not making more money. It's like, what is the growth that I have here if I'm not invested in the business, if I'm not learning anything new about the software, and I'm not getting paid more money, Like I'm just degenerating in every way, right, Like, so it's time

to go and what's really interesting? And well this doesn't offend anybody, But like when I see a resume and somebody's like I worked there for fifteen years, it's like I don't want to work with you, Like no things is like you you are so ingrained in like your Google you like I worked at Google for fifteen years. It's like, well, guess what, Like you didn't work in the real world, like you worked in like the perfect utopia

of like building software around other software engineers for software engineers like that. Like so like seeing that person that's like they have enough tenure where you know that they weren't just getting fired from job to job. They had that experiences, different teams, different stacks, different products where they can bring a lot more than just being able to type. Like that's that's kind of the value I'm

looking for. And I'm looking at a resume. I'm terrified if somebody just sat someplace forever, he just work on like the banner of Amazon for eight years, Like, what did you do? Right? So I know there's probably some really great time. Orry. I didn't mean yeah, I hear you though. That's me in a nutshell. Welcome high everyone right now.

I had a boss that I worked for a long time ago, and I I interviewed for another job and he found out about it was a smaller town, and you know, I was terrified that he was just going to fire me on the spot. And he said, well, did you get the job? And I said, well, they're waiting, are you? Are you mad? He said no. He said, if if no one else wants to hire you, why would I want you working for me? Damn, that's pretty savage. The funny thing is, I like, I've seen

this. The number of times is like a hiring manager, where like you'll make an offer to somebody, right they come in the interview, go al, I really like that person, Like I interviewed one hundred people or whatever,

and that was the one. I'm like, that's my person, and they're like my company counter and I'm like, yeah, if it took you leaving for your company to recognize your value, then you accept that, like I don't, I don't know, Like I mean, you take it, definitely take it because it's more money, but you should immediately reshot that out and find something better, Like take the money while you're there, and then go find the person that appreciates you for who you are and wants to give

you a good salary, not just because you're gonna walk right. Yeah. And the counterpoint to that is if you do stay with them, Like all companies go through down cycles, and whenever they hit that next downcycle, they've already set precedent that they're only going to do the bare minimum to keep you around, and so whenever they have to lay off people, they're likely to think, well, hey, this person has already been trying to leave anyway, so they need to be first on the top and block to go yeah,

especially if like you've like if that salary is no longer convincerate with the value that you have. You just got a twenty percent raise because you were going to quit. Like now you're the biggest line item on the team, right, But now I do think of this analogy just hit me. I kind of feel like what we do are like the wagon train masters of the Old West, Like our job is to get the settlers and their wagons out

west, and once we get them out west, we're done. Like you never hire a wagon master and say, well, we're going to use you to escort our wagon train for the next fifteen years. No, get us to this point and then go away. Yeah, it's kind of like it suffers, like we hump if you wanting eighteen eighty three, is that where this came from? Where you're just like a fore Gun trail fan. It's just a side quest finished eighteen eighty three is so good? Yeah, No,

it's funny. It's like, yeah, nobody's ever like, yeah, we'll get out the Oregon. We'll keep the wagging guy around in case we want to go to like Montana or Alabama or something. Maybe we'll want to go back to hudos, right, and like that's what you're just like, Okay, well I'm here until like we have another direction we want to go, we to change all this stuff. Yeah, that's an interesting one them.

I was talking to somebody the other day, a buddy of mine that's working on an AI code product, and I will call them out because I think the big disagreements on it. But like this whole notion of like we've had this like for like twenty five years for anybody who's newer to the industry. But like when Macromedia dream Weaver came out, I remember people being like, we'll never be like business people are gonna build all the web pages and

it's like, ah no they aren't. But like as like as right, because at the end of the day, like we are tools for business, right, Like so this idea of like no code and AI it can help this is people like build software Like that is a great idea, and like I'm sure that that will happen eventually, But in the meantime, us being relegated to being code reviewers for a machine like that feels like the most miserable

intersection of how this did go right. It's like the creativity part, and like thinking of how to like abstract like business logic like business needs and logic into code is part of the fun part. And it's like, that's just a computer and I'm just like, ah, I don't like the way you've spelled that variable, Like that's a that's a sucky life for software engineers. So but yeah, wagons AI, it's all the same. That could that could be our new startup wagon train dot a the bafest path to organ that

ended up. But there is there is like one of the things you mentioned earlier, you know about the the alignment of you as an employee versus the company. Like the founders of the company are vested in the product. Me as an employee, I'm vested in providing a really good living for my family, putting food on a table, putting a roof over their head, giving them the things that make them feel like they're having a quality life. And

so we have different goals there. But jumping around like we do also is a benefit to those companies because those people, like the diaper company, they're really passionate about how to get diapers to young families. But if I've worked for a bunch of different industries. I can bring perspective to them and help them build a product that meets that need better because I know the roadblocks that they aren't familiar with and can't see because they're focused on that the diapers on

the doorstep thing. Yeah, and it's interesting because, like you know, in those like as a company grows, especially from startup to so you say series V and beyond, like the product also changes, right, So, like one of the things that might bring you into maybe new e commerce companies, the person had experienced a previously e commerce compani It's like, oh, we needed somebody who knows inventory systems and you know, billing and all that

stuff. Like, let's bring that person in. But then as like this company evolved, let's say three years, we're doing like a lot of video options. It's like, why don't necessarily need like this person's deep knowledge and

e commerce systems anymore? I know they need somebody to news how to do concurrent video streams to hundreds of thousands of people that might be bidding on this item, right, and so like our business has changed, the people that we need involved change, and sure like that person that's been there for three or four years could go learn how to do like video streaming and you know all that stuff. But it's like, also you could go Hoch somebody from

you know, Prime or something like that. Right, So I think like that moving, I think is what makes us really good tools. Again, like if you sit in this narrow channel for fifteen years doing one thing for one product with one team, that may have people swapping in and out, like you were very uniform, right, and so like now when you're coming into another environment, I feel like it's harder to figure out like how to

how to necessarily fit you in, right. I feel like it was kind of like limits where you I mean, I guess when you have the prestige of a big band company can kind of go wherever you want. But like if you were like, oh, that person was at Macy's for seventeen years, you know like, well we got to get them in here, right, But it's someways like that it was a Google for seventeen years, You're like, oh, how much do how much money we have to live in

a bank. Let's give it to this person, give them the stock. Right. So when we were chatting over email before the podcast, you mentioned that DevOps is pure bullshit. So I don't want to care about this. Yeah, I'm gonna call you out on it. This is fine. I love talking about this, so, like I have to say that, this is one of those moments where, gosh, you can't remember who said the quote, like being able to hold two opposing ideas in your head at the

same time. Who was that? Uh huh. I'm not really good at quotes, and this is live, so I can't google it. But ultimately it's like a definission of being crazy. So I was like mesuring Franklin or something. It's like, oh, that's a sign of intelligence. It's like also inside of being nuts. But it's really I think it was it. It was one of these much smarter and successful to me said it, so you got to but it it is and it isn't, And I think it's

one of those things that's interesting interesting. It's like death ops is what it is based on, Like how much your zoom dinner is zoomed? Out? Right? You could watch somebody who's building an application. Let's say it's twenty thirteen and somebody's building a Ruby full stack app monolith and they deploy it on Heroku. They are they doing DevOps, I would argue, yeah, they are. They're they're operating their software happens to be running on a platform.

It's abstracted a lot away from it, but they're operating at the abstraction that they've been given, right Versus Okay, I'm an engineer. I'm sitting around three hundred micro services. I'm configuring the Kubernators cluster and all the service meshes. Like am I doing DevOps? Like yeah, you are too, You're doing like the dev and the ops like sure, So like it can exist at different levels and you can create abstractions where people can still do DevOps.

And I think that where we are today and how we like handwave a lot of the definition of DevOps. Like there's people that are in like it's a team camp. There's people like it's a collaboration camp. And there's the people that are like you do everything, like you build it, you run it, like you have to do it all. Ah, It's like okay, well, like we kind of like three definitions so like that right, there's a definite bullshit Like it gave defines, right, but like I have three

of those like don't happen for most people, And I have proof. We love the Dora report and DevOps, but when you go through it, fifty percent of people that respond to the Door Reporter like everything's fucked here and it's just like numbers, like it takes us we really software once a quarter and it's like okay, and it's like an outage lasts up to two days,

Like fifty percent of the Door report respondents report pure chaos. So like while walk like we're on Hacker News, We're like, haha, at my company, we do DevOps great. And it's like, okay, you're a soul CTO who deploys on verse cell, Like what you're doing it right, like your hands in fine, And then people will be like, we do devlops

great. We have eighty thousand micro services like you also have ten thousand ops people who've built an internal platform that makes it where you can do a DevOps with the abstraction it's given to you, right, And so I feel like I feel like like it is plausible. It is an idea that makes sense, and you do want your developers to have a baseline understanding of the abstraction

that they're given. But I think the bullshit is in like like like are we giving people the right abstraction, And I would argue, like, that's where it really gets super complicated, And the answer is also like we probably aren't right. So what's interesting is looking at another report. State of CD report is another interesting one. The most recent state of CD report found that only twenty seven percent of companies are using I see infrastructure's code, right,

and so we're all DevOps engineers. We automate everything, do we? If only twenty seven percent of companies are using infrastructure as code and I ventured, I bet most of them that marked yes and that aren't using it one hundred percent throughout the company. How are those other seventy three percent of companies that are doing DevOps automating stuff? If you weren't using antibal, chef care form or blooming, are you using Selenium to automate clicking through the AWS dashboard?

Like I'm confused here because we're all doing DevOps, but you're not automating any of your infrastructure management. That is where a whole other chunk of bullshit comes

in infrastructure management. Like some people consider it a part of DE but it didn't exist really, I mean it was like CF engine, but like there wasn't much infrastructure management or even cloud infrastructure when like the DevOps term was pointed, right, And so this whole idea of cloud infrastructure, which is this thing that ran our software that over the past ten years has slowly become our software, and it's going to get worse with AI and LLLM models, Like

a lot of our software is no longer ours. Right in two thousand and eight, my code was my code. In twenty twelve, it was like, well, all my cues and background processes are actually done by this sqssn S thing, Like I've got a lot less code to manage I call people other people's APIs and now it's like step function in land. That's like my code's getting smaller and smaller and this cloud service area is growing. And then you look at llms and it's like I don't have any code. I just

shove hashes in somebody else's database and magic comes out of it. Right, Like we're getting a lot less code, a lot more consumed cloud services. And I don't think we ever tradition like went back and like include infrastructure management like in that loom, right, And then when you do, whose responsibility is it? Because if you say, all of aws's the responsibility of all the services that you may or may not use in AWS is your engineer's responsibility.

You're out of your fucking mind. Like their responsibility is to build business value if they have to go read the twenty seven pages of how to monitor, configure secure YadA YadA, YadA, YadA, YadA dada postgress. When it's like locally, I just like Docker run postgress, It's like that's the abstraction that they should see. They should say, I need postgresses, and they should be able to assume that whatever's happened on the other side, I've

got secure compliance, scalable, cost effective thing. I shouldn't have to think about costs, and I shouldn't have to be sidelined on it three months later when the cfo's pissed off or we're spending thirty grand a month on a misconfigured post resistance. Oh sorry, So that's why I think it's bullshit. But

here's the catch is, like I'm also one of these people. It's like, hey, platform engineering, you got to do this right, which I will admit platform engineering is a subset of DevOps, but the catches, like most people don't have the budget to actually do it or the talent to actually do it because their DevOps team or operations team is in drowning and technical debt, and like trying to ticket ops for their developers and people that just have

developers and don't have an OPS team, they're probably better served by the past and going directly to the cloud until they get to the point that they need that scale. So that's me defending the DevOps as bullshit. So it was like a therapy session. Yeah, I'm gonna get your insurance details after the podcast here that. Yeah, so I look forward to comments after the show.

No, I agree with you. And the irony here is I actually have the title at my current employer of DevOps Engineering, and I agree, like that should not be a thing. And I think part of it just become to in my perspective is we've we've we're just kind of like that's where we've drawn the line in the sand, like, oh, we're doing DevOps, damn it, no matter how hard it is. And it's kind of like, you know, being a forward guy or a Chevy guy. You know, if you if you buy a Chevy, you're a Chevy guy.

Now you you know, it's like I can't buy a Ford, or they're going to kick me out of the line dancing club or whatever, you know, and we have this failure to adapt with the way that our our industry has changed. And you know, because you can't, like you mentioned, you can't ask a software engineer to be proficient in writing code and go, but you also have to be an expert in Postgress database and uh, an

expert in Docker and kubernetties and security gateways and cost management. There's just so much context switching there that you'll never get anything better than mediocre across the board.

Yeah, and it's funny because, like I feel like I might be wrong here, I probably am. Okay, Like I feel like two things have kind of got us into the mess that we're in with develops, right, because if you think about just like an engineer, like the amount of stuff that we will stack as a responsibility on engineers, right, why does this happen? Right? So? Yeah, I'm a software developer, right, software? Okay, I use a database. I picked postgrats. Okay,

well this is an Oracle nine I it's Postgress. So I'm assuming that you're the database administrator. Also software developer, you're going to tune your indexes right like a decade then fifteen years ago, like we had DBAs, but like you're seeing them become less and less prevalent in businesses except for like massive, massive businesses because like, oh, it's like that person will figure it out, and then so that happens. We stack that responsibility on people,

and I think it's twofold. I think one's business people. I don't understand like where the lines are. And we're again we're hobbyists. So it's like okay, like, hey, yay, will do you know how like LLM building works? And you're like no, And you're like, well, you've got a better chance of figure it out than me over here. It's working on the old it's right. And so you're like, if you give a software engineer like time to go discover something, they will do it. Now,

will they become a master of it? No? Will they get something that works enough to go into production? Yes? And that's where our product managers are content, like they're not like, oh, Corey, did you master building elements before you've lobbed this thing into a kuberinetes pluster. No? But it works right, And that's all matters. Like I made the button click faster, I sold more of the things. Like at the end of the day, that's what matters to the business, not necessarily that it's perfectly

secure. Do you care about that business? Do you care that it's perfectly secure? Or are you going to buy an insurance plan from Equifax to cover people's credit when it breaches. It's much much cheaper to buy the insurance plan than to spend the engineering resources to make it secure. Yeah. Yeah, as just ask my real estate and investment company that I'm working with, because I prompted them on that. I'm like, how much time you spill security?

Are things perfectly secure? Well, then we do investing. There was a breach and like we leaked your social Security number and all your financials. I was like, I knew my credit's locked already. It's fun. But so I think, like, you know, that's that's it's really interesting,

Like we'll just stack stuff on people. And I think that's kind of where it's easy for burnout to happen, right, Like you think you're going to be focusing on this problem, and you are, You're trying to solve the problem, but all these other things are kind of tangential to it, and if you don't have the people that know that thing, your company is looking around like okay, who has either the bandwidth or who do we think is like the best engineer that can go figure it out? And like now they

have another technology and other responsibility that kind of lands on their plate. If you're a small company, you're starting to grow and you're like, okay, like I'm doing a lot of this cloud stuff, like we should hire our first DevOps engineer, and so now it's eight people that don't have like this deep experience in the cloud. They are like, we should hire somebody that does. Then you start that job, you're the death first DevOps engineering.

You're like, okay, I had a lot of work for like three months and I was stressful, But now I'm sitting around like five hours of work to do a month. Right. Then that's where it gets weird, because like you can do two things at that point time. You can either sit back and be like I have thirty five hours of drinking coffee, or you can try to do like real DevOps how do I make this easier for those developers? How do I make things more efficient? How do I make things

more resilient. How can I start practicing some SRI principles since I have this

bandwidth to help make sure their software is staying up. And that's where I think I feel like a lot of DevOps people, particularly if you look at our slash DevOps, so that's a group of people not saying it, but like the amount of times people that are like I have nothing to do, Like I've been working here, like I work like ten hours a week and like everything's like kind of messy still, but like nobody's given me, like the DNS project to work on. And it's like, well, you're supposed

to be talking to your you're supposed to be doing that collaboration part, talking to your engineers for how you can make their life better, Like that is your job, and like that devlops role is like continue to make the software

better, the business better, and the engineering dev experience better. And I think that like once you start going down that path, like that is a good place for DevOps to be. But where we're just tossing stuff over a fence to a person and we're like, oh, well, the devlops team they configure the cloud things for me, Like that that's that's that's just two thousand and five. I'll just pay for sure. Yeah, I take the approach that as a as my role in develops. The software engineers are my

customers. So my job is to make it to like provide the tools that they can consume that let them do their job more efficiently and putting in putting in guardrails so that they can do what they need to do. But then I have guardrails in place so that they don't accidentally do something that they may not want to do, like you know, expose root access to the public Internet or something like that. Yeah, I think those guardrails are important.

Like that's actually a term that we use pretty pretty frequently and like our sales calls and even in our content, Like I feel like again like our software is becoming more and more cloud services that we're consuming and less of our code a lot of used to B calls, and I feel like when you're using the cloud, like there's so many services like how do you do how do you how do you do event driven systems on AWS? That's an SQUS,

is it event bridge? Is it POFKA, Like like I've had three or four options just in AWS, right, and then let's say I pick it. Okay, let's say I pick my solution. Looking through some of the AWS like models when you like post your thing to make a new thing, Like, some of them are insane. Like one of the most crazy services at AWS is that's three. That's three, like one hundred and twenty configuration options, right, And so there's a lot of stuff that is getting exposed

to your engineers. So if you're like, oh, I did DevOps because like I installed terra form and I get of action and now they can write whatever, Like you didn't you just you just gave everyone a massive footgun. You didn't make their life better. Sure they can do whatever they want,

but like should they be able to do whatever they want? Right? And I think that what we need to be doing as an industry DevOps, platform, engineering operations, whatever your company calls it, is like enabling those developers to do what they need to do, but also shrinking the surface area of the cloud. Right. So, like if you have an established operation scheme that understands all the cloud services, being able to have like a catalog of

like here's what we use for a venture and systems, it's COFA. If you don't want to use COFT, that is your choice, but it's now your operational burden. Like make an RFCs to why we should also support e vent bridge and like we will take on owning a vent bridge. But like if you're just gonna loan range it, like go for it, right. And also even when you pick a service, like you need to service, you need to shrink the surface area of that service, right, Like the

cloud's goal is not to be easy. AWS isn't We're going to be easy. If they're easy, they would only have lights fail. Their job. Their job is to be capable. Their job is that you can cut this pizza in any shape. You want to make your party happy. You want squares customers, you want we cut wedges, You want to do a weird little zigzag, you want all edge, I don't know whatever you want to

do, Like we should be able to do it with AWS. And here's all these things, right, But the reality is is like a lot of those configuration settings don't matter for your use case high Aurora, like you can configure Aurora. There's so many ways that I'm gonna go provision postcress and I'm gonna turn on this thing. That is this field that I can turn on. It's like that's only for my sequel auror it's like, oh, well,

then why was it exposed to me? I only use Postgress. I'm not going to randomly switch from postgress to my sqel one day, but I might randomly change a field name right, or like a field value right, So it shrinking that service area is important? Like why why? So? I would love somebody in the comments tell me why I'm wrong. Why is turning encryption off for anything an option anymore? Yeah? I agree with you, and I think there's like a certain amount of responsibility. I actually just

had this conversation the other day. I think there's a certain amount of responsibility from the cloud providers to do a better job here. Like if if I'm as a cloud provider, they advocate infrastructure's code, least privileged security on all

those kinds of things. So why in their documentation does it say, oh, if you want to push to the artifact repository, you need artifact repository dot admin privileges, Like do I really need admin privileges or was there a better way you could have phrased that to help me adhere to the principles that your marketing team is advocating. Yeah, and again like I don't, I

don't. Yeah, it's like I don't. I don't think if their goal is to be easy, it's just like it's it's just to have those capabilities and right like I AM is like a whole other just when you can see it, like it's funny, like I feel like some of people like what are you telling me I AM. It's like I AM is cloud blocking. Like everything we're using Dynamo CloudLock and it's like, no, you can use Scala. It's like S three. We're locked in. It's like I just

throw Minio in front of GCS or whatever. Like plenty of ways to get around cloud locking except for I AM. Like if you invest heavily, if you go beyond the asterisk, Oh my god. That is that is like everybody's im is just like completely different. You can you can tell how like hard it is, like there's I AM engineering roles now in large companies. It's just like, oh, like I can't like it's important. Oh for sure, I can't imagine. I personally can't imagine what that job. And

hear you saying with like pushing a container. It's like, oh, you know, I need right. It's like getting to those places where you're like, my sole role is to figure I am for this place, but the cloud doesn't give me the ability to make maybe maybe a particular service as secure as I want, right because of the granularity Google. Yeah, I like software. It might not sound like it. I actually love it, but I just yeah, sorry, No, I think that's part of it.

I think that's one of the common characteristics of software engineers is there's a really really tight balance of which we like more. Do we like writing a code more, or do we like bitching about writing the code more? And and it's it's a tight race bitching about it. I coulde. I still do a lot of development, probably more than I should, but I switched between Go Go Lang and a lickxer that's where our app's built in pretty frequently. And you know, whatever language I'm in, I'm just like, oh,

I hate this language. I wish it's if I'm going to rewrite this in language acts, then you get it done and you're like, that was stupid. Yeah. I had one of those yesterday. Ask him about four hours working on something, and I was just like, I finished, I was like, why did I? Why did I? I'm had a purpose when I started, it got way out of hand, like I still do it, and like I over engineered the ship out of it, and I was

like, I don't. I don't know why I spent four hours on that, But I gotta go do some real work now, right, Yeah, yeah, I'm gonna do something that I can actually claim because I can't show tomorrow and say that's all I did. If you feel that way, people, it's hard to shake. This is this is This is me years into being a CEO, and I will still go over engine or something if I've

got a spare of time. Oh for sure. Yeah. I actually for my side projects, I have a group of guys that I hang out with, and we all know that our sole job as part of that group is to call each other out on that specific thing, like are you chasing a shiny object? No? Well okay maybe yes? All right, fine, I'll go back to what I'm supposed to be doing. I'm gonna start the antithesis of that group. I'm gonna start like a Rube Goldberg engineering group.

I think you've made that more complicated and like throwing some more technology into It's like, you know what I could, I could I'm gonna spend a couple of hours this week and doing that an LLM that generates Bash scripts. Oh man, that would be just go. I'm afraid of I'm like the next five years. I'm like, oh, I'm in like like a I think I'm in the trophic disillusionment around AI. Like people keep asking us, we did our fundraise this year. We raised eight million dollars and we did it

without putting I mean, it was a hard year to raise. We did it without putting the words machine learning, data science or AI in our deck. And we've stayed away from it in like our materials just because like I don't I don't want to build our product off of like some like trend that maybe not go where we hope it goes. And seeing some of the other people in the OPS space like ah AI ops and You're like, Okay, I'm going to give this a spin, and I'm like, I would like

a Hippa and sock two compliant Ubernetti's cluster. And I will not mention the name of the other cloud provider, but I shit you not. The code that it generated was some type scripty codeink wink. It was all the code to like initialize an KAS cluster, and then the comments I kid you not, I kid you not. The only code was like some import statements. It was like stree equals new EKS cluster, and then in the comments it

was like configuration for stock too and nip A compliance here. And I was like, yeah, the intelligence is pretty fricking artificial if you ask me, but like I don't know, like the thing that's really freaking about it.

I kind of got it, couldn't even imagine it for bachelors be terrifying, But like this is this is again a part of our problem, like we don't have enough opps experience in most of our engineering team, and so the idea kind of like business people looking at engineers and not being able to tell like where the lines are the same thing happens when you go and ask something to write software for something you're not familiar with, if you aren't familiar with

running Gubernetes, or maybe it's even just configuring some serveralist stuff, you know, step functions or whatever, and you're like, oh, I need some terrorform. It'll do all the step function stuff, make it for my computer. It's like okay, like cool, you might run that and it might apply it may or may not do what you intended it to do. But like now you have a bag of code in front of you, you don't understand it's doing something. You don't know if it's doing it right. And

like chat GPT was confidently Hey I did it right. I think like you have no idea, so like are you going to go Google every single one of those lines of code that isn't the resource block to figure out like, oh, is this is this what I need to set for this? Like no, you're like just like the product manager looking like that works right right? Yeah, I got step pumption. I'm moving on. It's like that's

that's freaking. It's like I would rather have you just gone on, Like I would just please go click it in the AWS console, like turn on a macro reporter, install the Chrome extension that led like records your mouse clicks. Go do that and just use the Chrome extension as like your I C tool at that point in time. I'd rather do that than run something through AI and hope it right. Like that's great. I've been I've been struggling with this for a while is for people who do click ops, how do

you how do you document and track that? But I never thought of just using the click recorder and Chrome. That's brilliant. The problem solved. Yeah, yet we've got these We've got these video files. Just watch them and you'll see how I was built. Well, if I'm a lot of the I'm not sure I'm a lot to plug myself. But you're looking for something that feels like clickops through engineers, but you want all that operational maturity and

security. No, I do want to talk about that, which is one of the reasons why I was looking forward to having you on the show here, because I like I'm at the point where I advocate strongly for infrastructure's code because six weeks from now, whenever something is different, I want to be able to look at a get history or poor request history and see what has changed, or run terraform and see that whatever's running out on that provider doesn't

match what terraform says it should be. And that's been one of the big challenges I've had with the platform engineering tools, that some of them are just clickops on top of clickops, you know, and it's like if I just want to click ops, I would just use AWS. So how do you approach that? Yeah, so I think that. Okay, So I'm gonna

tell you a couple of things about me. I'm a software engineer, so I understand like I understand vendors, and I have to struggle as like a CEO and a founder and also a developer where like I don't want to be the annoying vendor. That's why I was like, I don't know want the plug like I want to talk about like I want to sell this thing,

but at the same time, like I don't want to annoy people. But uh but yeah, So what's interesting is so the second thing about me, I'm going to say something and I'm going to be I'm not going to be the founder that says this and is disingenuous. I think we're going about enabling platform engineering much differently than a lot of the other tools out there. Right, So what's really freaking me out about this is I think platform engineering is

important. Actually, I make a note of like the clickops, how we do it different. I'm gonna come back to that in a second. Do I haven't have time to set way? He said, we can go on forever, right, I can, Yeah, there's there's there's no time on it. So here here's why this is important. I spend a lot of time like reading surveys. I like it. It's one of my weird things.

And I feel like a lot of times, like when you get a survey at the door report, there's like a couple of like pull quotes and some numbers, and you're like, oh, we're all doing much better. But then when you like get to page like thirty stuff and you're like, holy crap, everyone's gonna die. Looking at the stack overflow survey for the

past four or five years is really interesting. If you look at cloud operations experience, every every administrative role, operations role in the stack Overflow survey has steadily declined in the amount of people doing it over the past five years. DBAs, SRES, DevOps, operations, all of them are going down. Now are we retiring beast? Maybe we have a lot of grave yards that could be the reason. But also we're just making engineers past. We're making

engineers faster than we've ever made engineers ever. Remember seventy years ago, there was like one software developer. Now there's more software developers on the planet than there are people in Australia, Like you met an entire, huge ass country full of kangaroos and software developers. Like that's something we could do, would be really weird, but we shouldn't do it. But like right, so, like where are these people coming from. They're coming from boot camps.

Like we're still on a lot of people coming out of universities. We're printing people out of boot camps, and that's not a bad thing. Some of my favorite engineers I've hired are out of boot camps. There's a personality type that I look for when I see somebody coming from a boot camp, like a fresh grad no history, that's eighteen. That's a person that was really those probably doing software, got very excited, went straight to a boot camp

because like I want to get in the job. I don't want to sit four years learning about algorithms to build a database, like I want to build a website. That person, right, the person that worked at a power plant for fifteen years and they're like forty three, that decides I'm going to give up on this career that I've been doing all this time to go in software, Like that person has grit. I love that person. I think that boot camps are an important part of our society. Why how do you

attige boot camps to like civilization? Everything's moving to the cloud. Everybody is a software company. We're becoming more and more dependent on this stuff. We need more software developments. But what's happening is boot camps don't teach the cloud. Boot camps don't teach DevOps. They teach you to build react code. They teach you to build rails. They teach you to add business value as quickly as possible, which most businesses don't see. DevOps is business value.

Right. So it's like you come out of this boot camp for twelve weeks, you got your twenty six thousand dollars tuition with like eight percent or whatever, and you're like, I'm ready to write some reacts. I'm ready to rails, generate an active record model, like let's go. And so those

people come in and they don't have any DevOps experience. The don't have any operations experience, right, And I think that's where we're seeing this, like like this steep drop is like some people are retiring because like we do have a lot of gray beards, especially on might be in the data centers and more on the administrative like Linux to step inside. Those people are going away for sure. We're also just printing a bunch of people that are very much

closer to the top end of the stack, right. And the thing that sucks with OPS is we don't teach it in universities either, because if you talk talk Kubernetes. Actually, if you if you rewind four years and you taught people how to deploy in the cloud, what would you be teaching them. Probably not Kubernetes four years ago. Today it'd be Kubernetes, but four years ago maybe the two instance. Yeah, maybe, Like nothing is really

I'm gonna say something bold here. Nothing's really changed about our software in the past seventy years. LOOPS variables ifs sure, you've got different syntactical sugar on it. But in the last twenty years, everything about the way that we

package and ship software to the cloud has changed four times. We had data centers, then we had vms, then we had everybody started going the servi list, and then containers came around, and then like big Container orchestration, then servilest containers, now llms, like we've changed the actually llens might be the first thing that's happened in like the way we write software that's fundamentally changed. Besides just like if look, you're not because they're all just jam and

hashes into somebody else's database. Now sorry, So so that is sorry. So that was me explaining like why we're doing things a little different, Like the the amount of operational knowledge on this planet is going down in relation to the amount of software developers we have, and those software developers are only going to learn OPS on the job. That's a problem, right, Yeah,

are they going to learn ops from? Are they? Are all of your DevOps engineers going to turn into You're going to make a fifty room campus so you can teach all your software developers like how to do a DevOps You have brown bags every single day. Like you're not going to transfer that knowledge from your DevOps team to your engineers in an efficient way, right, So you

need to build the substraction. You need to lower the surface area so they can do as much as they have access to without having to like jump over that wall. Right as soon as you reach around that wall, you're like, hey, I was able to deploy everything myself, but like it's down Okay, we they gotta be able to troubleshoot themselves, right if they have to come around to you and be like, hey, I deployed this thing

that build a container. It went up and that is just fucked Like if they come around to you like that, like they need to have that knowledge themselves, like we need to be building those abstractions they can solve these problems

themselves. And also at the same time, I think as a platform and we do this, and we hope that our users do this for their users do this, but like as platforms and people this knowledge, we need to figure out how to get it as efficiently as we can into developers because we do need more people understanding how this stuff works. And I love when one

of our customers even is like, hey, how does this work? Like under the hoot, It's like, yeah, you should you should be asking me that, like you should want to know that, but like if you don't, you should be able to do your job effect And so what I've so this is a long lead up to me saying I think we do things a bit differently, which every startup founder sets but the thing that made it different for us is everybody's talking about platform engineering, and the first thing they

jump to is who there's like Kubernetes. Oh, guess what, I love Kubernetes. I develop on Kubernetes locally. Like that's how much of a family I would be, Like, if you want to use my product, you have to use Kubernetes. I just made a choice for you, right, Like you might be perfectly fine running on Lambda, right, but we want

to use that forever. So I think that, and I think one of the things that's a little scary about this idea of platform engineering is like we're leaning too heavily towards Kubernetes and not accepting the fact that there are other things. I mean, not that anybody's going to run out and run masive sphere, but like there is nomad, there are other run times, there are things that are outside of Kubernetes. Right. It's very plausible that your software

teams in your company operate their software differently. And so the idea that you're going to come in and be like we have one platform and everybody, nine percent of you are going to fit your code into this one platform. It's like, Okay, that sucks for the jam stacked people. That's suck inc. So what are we doing? Are we gonna run Are we gonna run llms on versell? Are you gonna run jam stacked on open Ai? Like?

How are we gonna fit this all on the one platform? The reality is like much like organizations are different, like your teams are different, and so you can't have a platform that does everything. Those engineers need to be able to have the agency to get what they need from the cloud without constantly bothering another person or reading a white paper, and the business needs to know that that software is secure and compliant. That's hard to do when most companies

are abysmally failing at DevOps. Fifty one percent of you based on the door or report, So if that's not you, one of your friends is lying.

And also like a lot of these companies, like you can see a company with like fifteen engineers and like nobody really has ops experience, right, So it's like we just don't have enough of this, and so our approach has been to focus on operations engineers and DevOps engineers that are either coming to this realization that they cannot scale themselves or their company is kind of mandated,

like hey, Gartner said, eighty percent. Companies are gonna have a platform team by talking about the wrong side platform team by twenty twenty six, and it's going to make DevOps more efficient. So DevOps Guide, you're a platform engineer now and does that mean right? And so our approach is we should be able to work with So like our product's number one user is operations,

they don't use our site, they use the tools that they know. So mass driver today works with Terraform, open TOFU and helm and we're currently working on CloudFormation and Gloomy support. And so the idea is you as an operations team can decide what cloud services you want to support and then you limit that API of those cloud services using Terraform, Helm, open Too, Food, et cetera. And so you just publish okay we or Kafka for events,

we support postgresses or transactional database. We run on Kubernetes. We also support land n API gateway for the services team, and you essentially write Terraform modules with your whether it's OPA or bridge crew, whatever inside of it. You create these modules that are your businesses, security compliance and practices spotified into the naming conventions, et cetera. You publish them into mass driver instead of like

the Terraform registry or whatever. And now when your developers come in, what they see to state instead of seeing all the services of a WS, they see the services that my operations team supports. Now what's really cool is they just draw infrastructure. So like, okay, like we run on Kubernetes, Like, throw my Kubernetes cluster up there. I have a whole thought, I have a whole I have a whole rant. I can go on about how many Kubernetes plusters you should have. It should be more than one.

Cattle cattle versus pets. People, you got all these all these applications or cattle cluster is certainly a pet. Is that you're afraid to change it. But we named it and everything we actually had, sorry i'm gonna say something, and a little side tangent. We had a cople of company flip trips that we were at. We had essentially two Kubernetes clusters of van production.

And what we would do is every time that we were doing a Kubernettes upgrade, we would do it too essentially our standby, and then we'd roll applications, cut traffic to it. And so like that's how we kind of determined that, like okay, upgrading at one point two seven or whatever, it worked. And both of those clusters were pets ish we created them with cub admin or cub no cops, but they both had names and it was because Siri could never get Kubernetes right, so we had cuban Eddie and Kubernetti.

But anyways, still like our our approaches, all right, I'm a I'm a brain map to follow me, but so approaches like make sure the operations people don't have to learn a new tool chain, right, like use the stuff that you already know, Like there's not enough of you. I can't as a business, have you spending more time learning a bunch of new stuff when engineers are already depending on you for stuff, demanding stuff from you.

Your team is already stressed out, and now we're saying, hey, we want to make we want to scale you to make it more efficient for them by adding more work to your plate on top of the other stuff that's going to happen. It's like okay, well, or was if we're also kind of like helping with outages and monitoring, et cetera, like Are we just not doing that anymore? Why we go build a platform, right? Are we all going to turn into React engineers so we can build backstage plug ins?

And it's like no, Like so our approach has always done an operations engineer, Like we wanted to build something that worked with the tools that your team knows. You publish it in there. Your engineers just draw the infrastructure, and so as they're drawing, what happens is they drag let's say they drag an API gateway out or a lambda. There's these additions that you can make to your infrastructure's code we call we internally we call them artifacts. But

it effectively lets you create a diagram of your terraform. So you can say, like my application run time like terrifle module or healm module depends on Kubernetics and so now like when you drag that into the campus, that will automatically connect like the nearescubernets cluster. You can move it around like your engineers don't have to think about it. So it's like, Okay, we're building this service for the first time we wrote it. I'm going to use the helm

chart here for running going applications. I drag it on. I set my Docker repository and it just connects to kubernets cluster, I hit deployed and I get back to work, right, And so they have this limited view of the cloud. It's the services that your team supports. But then also your terraform variables are also when they come into mass Driver, are become heavy documentation.

So it's not just like oar name like. You can actually mass driver supports like super rich and like cross validation, so you can say, like if the name is this, then this other field has to have this specific validation on it as well. So you can start to codify a lot of constraints into the system. So you can say, okay, we have an Aurora postgrest bundle and it can run. If anybody's a familiar with Aurora,

you too. There's an instance type called serverless. It's very weird, but so technically you can switch your Aurora back and forth between servilists and server full. Like you experience a downtime while it's happening, but you can switch it

back and forth. Right, But the configuration changes wildly depending on if the instance type it is serverless, not if it's a servilists Aurora the instance type of servilist, right, and so like internally we have an Aurora Postgres bundle that we use and we use servilists for all of our not like our preview

environments, and then we use to reful for our production environment. And so that configuration like it opens on the side, panels your terraform variables and then as you change values, the configuration changes it, it'll actually expose and hide fields based on this. So now's the operation scheme. You can say, you know what, I know that this my sql bin log stuff of Aurora

doesn't apply to Postcrest whatsoever. So those fields that there's no those variables are even in here, you can and worry about thinking about configuring your resource. And you can also say it when they you have this men in max scale for your instances, when somebody selects a servilist instance, those fields are hidden so they don't accidentally say I want ten serviless and instead they show it shows you acus right, and so you can start to codify a lot of convention

there for them. And so now your engineers they don't have to learn a new tool leaver, they don't have to go copy and paste a bunch of Terraform modules and do a directory, set up some GitHub actions. Literally, the only thing to do is they put our CLI in there and they says, Okay, I want to deploy my production database or I wanted to pull in my production application, and it's pretty much just like the name of the

resource and mass driver and it deploys. You can put adjacent file in there that it'll get applied to either HELM or Terraform or flu Me to configure it based on the constraints that you've defined as an operations engineer. And now what's happened is you're able to scale this operations team. You can do platform engineering effectively. Mass Driver we're a platform orchestrator if you're familiar with the seven hundred terms that people have invented in this space, and each of your projects and

mass rivers effectively its own internal developer platform. But the cool thing is it's not developers having to operations team doesn't have to go and talk to developers and make an MVP and make a giant form to survey them on what they like and don't like, and collaboration back and forth. You say, this is what we support as a business. This is all that we know. This is what we're using today, and if you need a new service, we get hub issue and we'll build the module for it. It'll be a mass

reever. You can drag and drop it. Right Like, at the end of the day, all of your modules and mass rivers get up repositories where your code is like that's how we kind of do the collaboration. And then your ops team didn't have to learn anything new. Your engineering team didn't have to go learn terraform or polloomy or gett actions or anything like that. They can focus on writing code. They just drag something onto the canvas when they need it. You as the ops person, get an audit log, get

deployment histories, you can roll back. We have automated costs and metrics built into it, right, so trying to give that full agency to the developer. As soon as you can figure something a mass driver, a day later,

costs show up. So if you can figure this postpress citizence, you're like, holy shit, it's three hundred dollars like the next day right Like, okay, I'm gonna change my configuration here because I can also see utilizations three percent, so I don't have a mad engineering manager mad CFO three months later when they realize that cloud goes high, I, as a software developer, have this agency to control my costs. Do the same tool that I

provision it in the same place that I can see my metrics. If we turn your dash your canvae, you're essentially your your network diagram is also your dashboard. So you can say, okay, I want all these metrics from cloud watch to show up alongside this instance and set some alarms for them. So now when you get an alarm goes off at two am, because that's when grimlins use the internet, right, I think, So it's like two thirteen, it's like, what in the hell is going on? Who is

a Crown job schedule right now? So now, like at two am, when you get an alarm in an email, it's like you would like Peter do right. You think about where we are today, It's like, oh pat patre alert and you're like, oh fuck, I hate that sounds two thirteen in the morning. Click on it, uh Ari zero alarm and data Dog. You go to data Dog and it feels like you just walked into Vegas at three am, just like crash all over the ground. You're like,

Okay, some something's broke. I can see the things going up this way. It's red. That's not good. But like, what what's broken? Okay, it looks like I'm out of database connections on my application? Well, shit, like our traffic's no different than normally is I din't the application hasn't been deployed in three weeks because it's pretty stable app. We're not

like actively working on Like what's changed? Like if the traffics the same, the application hasn't changed in weeks, Why am I out of database connections? That's super weird. Well, the reason you're out of it is because Tony on the ops team, he got in trouble because the post residence was too expensive. CFO was mad. They're like, we got to control our costs. He scaled down the post for instance, and it can't handle as many

connections now. So now this change that happened that's not in your purview as a developer, and a different git repo has caused an outage on your application, and you're sitting there trying to figure out like, why why can't my application pack the postgress? If you don't have access to that, get repo. You have no idea if you don't have a dashboard for that Postcriss sentstence, you have no idea. If you haven't seen the cost to change that

post, for instance, you have no idea. Like we don't have agency as developers through chaos. Right, So now you're sitting there for hours trying to figure out what it is. You hop in the slack and you're like at channel like, has anybody changed anything about the data it's And Tony's like, yeah, yeah, wait, there's some costs of things. It was sitting at like ten percent CPU all the time, so we just scaled it down to some T three micros or whatever. And they're like, oh,

there's your problem. Right, And so with mass drivers with the story's a little different. Is you get your alert, you click on the link, we take you to your infrastructure diagram. We highlight the thing that is broken, and from there you can see the metrics and you can see the change history of everything. It's the dependency of your applications, is your cluster. You see your databases, you see your cues, and I can see right there. I'm like, oh, Chris made a change to the database yesterday.

What did he change, you click on the database you visually diff will show you the difference to actually keeping track of the changes in your terraform Holme et cetera. It's like, since yesterday, this is what's changed the instance type. And you're like, there's your problem, and now I can fix

it. From right there, I can go back to bed and in the morning, I'll call the CFO and be like, that instance is going to be expensive again, right to my app to deal with that, maybe if thro a PG bouncement, then the mix, like the ability to like get through and resolve an issue like this was never like our intention to the platform.

Our attention was like letting people get infrastructure securely and safely. But what's happened is like it's just much easier to understand what's going on when you're looking at a diagram, right, Like we all want diagrams. We draw it on a whiteboard, we go to our desks and we do some crazy shit

and terraform. And then somebody comes along six months later and they're like, to diagram this, They terraform, visualize it and get that horrendous resource bag, and then they try to back that out to like a lucid chart or like cloud Craft, you're like, hey, we've diagrammed it and nobody updates it for six months, and then six months later somebody someone breaks and you're like, this isn't what our system looks like at all, Like we want

it to do. Anybody who says we don't want it to be visual, do you've ever seen touch a whiteboard or cloud craft or lucid charts, You're a liar, Like, we want it to be visual, but like there's not a way of doing it. So I think that's what's fundamentally different about us, Like we don't want to just throw a portal on top of Kubernetes and be like you can see all your services. Like it's like we wouldn't change the way that you think about managing it and change the way to think

about troubleshooting it without making you change all of your tooling. Like developer in this scenario just writes code and your engineering team or your ops team says, hey, we support on AWS, we support ekas, and LAMB does great. We have two separate platforms to land the platform in a Kubernetes platform and mass driver. I use the one that my team needs if I'm on the back end team and we're like, actually, you know what, I'm making

the service. I don't know that need a full instance. It is very small. I could be able to run on LAMB, but I don't have to go and try to get the ops team or the platform team to like build out this new functionality for me. I just said, we already have API gateway and Lambda. The front end team uses it for some jam stack

stuff. I'm going to drag this LAMB over, push a small function up in there, and call it a day, right, Like, like, why should I have to engage with a product team and spend time trying to convince them of what my needs are waiting for them to build an MVP survey. The rest of the team's like I need to get some work done, right yeah. So yeah, because then your your ops team becomes the bottleneck and the limiting factor to your business. And that sucks because like I feel

like we're always the bottleneck. Yeah, they're always the bottleneck. I mean nobody's ever like, oh my gosh, I was going to go ask the ops team for something and they already imagined it and did it before I got No matter what even in the best scenario, you're like you're waiting a bit, like we're always a little bit of bottle. You bring us in early on a project and you're like, Okay, we're working on this project.

We're we're trying to figure out what the cloud infrastructure is. It's like you can do that, But I also feel like when you're very very early in a project talking about what it could be, like you don't know how the software and requirements you're going to change, right, and so like the end of the day, in reality, it's like any given terraform resource or collection of resources isn't months of work for us, right, It's like trying to

figure out what you need, what abstraction to expose to you, and then I cobble up some terraform, put in some practices, right my OPA rules, and like off we go, Like I don't need three months to do that for a couple of services you're going to use, So like when is

the right time to bring me in? Probably like some percent of the way through the project, Like once you have something like viable running, maybe you're using local stack to like figure it out locally, or you're using mass but as you can, but so like like when do you bring us in, Like we're always a little bit of a bottleneck, and the reality is is like maybe Google or not because there's ten thousand OPS engineers, but like the

average company where you're out ratio, I think we say like one to seven is like the good ratio of OPS to engineers. There's not one to seven on the planet. Yeah, but like if you look at the if you look at the stack Overflow survey and you look at the numbers, they're like it's it's not there. Like if we have twenty seven million software developers, we do not have two points. Sorry I'm not good enough, but we don't have seven million operations engineers, right, Like, there's not that many

roles. There is that many roles and they're all sitting open, right Like we have one of the longest roles that sit open. We also have one of the shortest tenures. Why is that? Why does it take forever to place with us? And we quit all the time. We're also one of

the highest paid people on the team. It's because we bounce those jobs constantly, get those raiss Like we are always going to be a bottleneck until we start doing something that helps us scale, and even in the platform engineering scenario, like we still are a bottleneck when somebody starts to move outside of the platform, right, right, we get that good balance of like the platform, like I'm never a bottleneck to you because the platform starves what you need

and then when you kind of go outside of like the norm, you go off that golden path. Now we might be a bottle that now we're talking about being a product. Okay, great, Like what do you need, Like, how can we do this for you better? Okay, great? We need to support ECS because there's some ECS task integration with redshift or something. I don't know. I don't think that actually tips Like let's say there's like a very specific use case like okay, cool, let's put together a

real proof of concept terraform module that has ECS. I'll push it up to

you. You can throw it onto your canvas, connect it to the redshift module we already have, and like, let's let's iterate on this and get it good enough so that we can have like market like production quality for any other team that might need to use in ECS, right, And I think that gets into the social aspect of DevOps too, Like if you can socialize to the teams that you support, these are the products, like, these are the platforms that we've built for you to use, and you can like

define the rules ving engagement. Then they know when they're venturing outside of those rules engagement of engagement, and then they need to come back and talk to you about that. But without doing that, the engineering teams see DevOps as everything being one off. They're like, oh, I'm going to deploy this thing. I gotta go talk to the DevOps guys, and it's such a painful process that they wait till the last minute because they just don't want to

deal with our bullshit. And so I think, just like creating that catalog for lack of a better term, like the here's the things you can choose from the catalog, and we do custom orders. We just need a little lead time. But if you can like get get them to that point where they feel like they understand what you do, then you can socialize that conversation.

And I feel like those conversations are always hard, right because like I feel like a lot of people again like maybe not everybody that's listening, like has this experience, but like when you go to an ops person, whether it's a ticket or you walk over to their desk and like, you need something that's like out of the ordinary, right, Like let's let's let's imagine that you had this magic tool where you can actually find all your terraform modules

that your company supports, right, and you go to that, you go to that repo and you're like, ah, like Mango at list isn't there. We need Mongo outists, We need a document store. Maybe they don't have maybe they don't have postcrests and they don't realize that you just use postcrusts. Sorry, I need Mongo at lists but it's not here. Now, I got to go talk to the ops team. Ticket got to figure it out myself. Maybe if that's our if that's our culture, and that's fine.

But you go to the ops person you say, hey, like we need Mongo out lists with a new project, and it's like we do you need Mongo at lists because they send you some marketing materials? Or do you need mongo because we can throw Mango and a container for you? Like what do you like? Why do you need at lists? And also like important question like why not why not document dB? Yeah? In aws? Right, like and the catches is like it's really funny. Like people anyone out

there is use document dB. I know, I know, I know, chill out. I'm trying to make an example here. I'm I'm not a document dB fanboy, but like reality is like there's a like document dB like can have a pretty similar experience to Mongo Like there are places that like obviously Mango's going to move faster in feature development and like the Mango space, yes, but if you're not using you're just using some basic document store, like

you might not need everything that Mango has. And if it is, I think Mango even publishes like a last time I checked, it was like either like a list or tool that could show like what document dv didn't support. Right, They're like, hey, like use us because this doesn't do this one thing. It's like you don't need that operation now and you can't see

the chance you need that operation or configuration option in the future. Does it benefit us to spend the money to have an enterprise account or whatever Mango atlas where we'll bind to our vpc. We want to go through the effort of peering our VPC with Mongo aut lists, Like do we want to go over

the public internet to hit them. Like there's a lot of questions in the mind, and so like the gut reaction from the person is like why do you need why do you need Mongo Outlets, which like it's very hard to ask that question without sounding like you're criticizing the person that's made that decision, right, And it's like, I'm not saying you're wrong, I'm just trying to understand, Like I don't know if you know all the options that are

available, and I'm trying to figure out, like what is the right service for us? Right, so, like can we can we try document dB so it doesn't create a ton of work for us because it's if you need document dB, I can throw together that terriform module. I'll skip lunch if

you do this favor. But if you need Mongo Atlas, when we have to go sign to enterprise account, talk to procurement and do some VPC bindings, maybe we have maybe we've made the dumb choice of having you know, that that side or something that that makes it impossible for us to buy into other ords it to appear to other people's networks, right, you don't know, Like you don't know like how well or chaotic our VPC is, right, So like there's there's other things that come to mind when we start talking

about using these. They says services that sit outside of AWS, and so I feel like, like that that's just like a problem that exists like in our in our collaboration, and I feel like that's another thing that like we just need to be able to simplify, right, like being able to say, like, these are the things that we support, you can quickly find them, right, and be able to get new services into people's hands quickly even to experiment. But like how do we get the time for that?

Right? Like I've had cases where people have asked or something like I could throw together a document dB. Like this is a real scenario. I'm parroting from a previous job. It's like I could have put a module together for them in maybe an hour or two for them to try out document dB to

see if it's good enough. But I had too much work to do, Like I didn't have enough time to do a little experiment to see if I could save myself a bunch of time, a bunch of time, right, And so it's just like okky, like they need this thing, and it always seems like they needed now right. It's like, oh, like why didn't this come up. It's like, well, we didn't know we need document databases when we started the project. You were at that meeting. I

was at that meeting. I wasn't at subsequent ones because like there wasn't anything that was needed for me or using postgrass got to the point where like somebody's like, oh, we actually need Mango for this, and someone brew installed Mango or they added it to their doctor composed file and they didn't think, oh shit, somebody else over there is going to have to do work right, So now I'm a bottleneck again. It's like I'm not a bottleneck.

It's that whole like, you know, lack of preparation on your part. It's not an emergency on my part. Like an asshole for saying that, right and sure, so like that's the thing that just kind of sucks, like that collaboration is really really hard to do well for most organizations. Can we jump to that thing that we were talking about earlier before the call started? I want to share this story. Can we talk about do we have time to talk about some IT stuff some some IT help est data stuff.

Yeah for sure, because we both we both started our careers in the help desk. So help desk corse stories. Put take one, go for it the one I wanted to share with you earlier. So my career. I was originally a physics and an architecture major, but I was working at a hospital and I was working in data entry. I'd been doing software development. Was like fifteen or sixteen. I wrote a little program that did my job.

I did my data entry, so I just like read from Excel docs and like input them into this like terminal system used for explanation of benefits for the hospital I worked at. And when I automated my job, I wasn't a software engineer, so I didn't sign like a PIA forum. I didn't. They did not have rights to my software. I wrote it at home like it was mine and say buy it from me, which is exciting. So how I got into help desk was they offered me a thousand dollars for

my little program and a job on the help desk team. And I was like, what, dude, pay me nine dollars an hour and give me one thousand bucks. I'm eighteen. I'm like I'm rich. I literally I took that thousand dollars, Like did you not I put a down payment on a wave Runner like I was imagine I didn't have a place to park it. I did not think this through. I lived in a dorm. I'm like, what the hell am I gonna do with this wave Runner? But

I was just so stoked. I was on the help desk team, and wow, like help desk is like we give those people so much shit, Like I think of every support person at police. It's like it's so hard not to be like especially like especially calling help deskers anything technical as a software engineer, like yes, I know, I understand how to log and just

yeah, I know, like right, like and so it's frustrating. But like those those like we're frustrated with them, but like their job is really hard, like they do run into a bunch of people that are just not familiar with computers all day long, and so they have to assume like they're

meeting more people like that than people that are right. Right, we had seventeen offices and my second or seventeen offices across Florida, and so like, aren't my my domain was pretty big, like to drive places all the time, I had a couple of like data centers, like it was the first one was in the building I was in. We have second one up in uh, like Zephyr Hills, and so I'm in Fort Myersporters. So anybody in Florida, Fort Myers or Zephyr Hills, imagine driving this distance for help

desk golf. So next time out that person there talking to you like you're dumb, like this is this is their life. This is a multi hour drive. And effectively what has happened is like there, gosh, I can't remember what it was. Something was frozen on their on their computer, and every time they rebooted it was frozen, and it was just like okay, that's a heart and the troubleshoot, like I can't even like remote desk into

it, like because they rebooted it. And this is, by the way, this is two thousand one or two thousand and two, so we didn't have the great tools we have today. So I drive two hours to Zephyr Hills. I go into the office and like, show me this computer, and sure shit, I sit down at it the screen, I grabbed the mouse and go to click to log in. The mouse I hit the tab button and it tabs into the username field and I'm like, it's not frozen.

That's interesting. So that I put in like one of the AVON passers. I log in and I'm like, okay, I grab the mouse and move it again. Nothing and I'm like. The first thing I do is I look under there's this big tower. The mouse is unplugged. My ve driven two hours for an unplugged mouse, and I have to drive two hours home. And I looked down and I'm just like and so I plug it back in. It works, And she was like, how did you fix

it? I'm like, the mouse, the mouse, and this is this isn't like a USB mouse is like one of those old like DIN connector things like real hard to yank this thing out. So what I was able to tell from looking at it as well was it was pulled out with some force. So the assumption is it got wrapped around her rank ankle and pull out of the machine some of those sliding things right here, but it slid out with like the keyboard right in the mouse right like there was some serious force

pulled this thing out. Someone must have known, like I feel like a trick must have happened. But yeah, that was that was one of my that's one of my favorite ones. That was a little fun. My favorite one that was not fun. This is second second. I ended up kind of focusing in healthcare once I changed careers. So my master's is in healthcare information system, so work for another company. And we had these two offices that were gosh, I don't want to give away, Well, I guess

I have to give it away. These two offices that we're connected to the data center here and literally across the street was one of our heart surgery centers. And this is two thousand and three or so, and like we're pretty far along, Like we've got Wi Fi all over the place. We've got a like a digital pills system for dispersing like pills in our pharmacy. Like we've heavily invested in technology. We had a really great engineering budget. We

had badges blogging the machines, which is a nightmare for him. We got to find so much from doctors just leaving their batch's places by like a hip a shopper who came in and like the doctor left the card and they logged in and just accessed a bunch of shit. It's just like kill me. But my boss was, this is like in the days when you could haze new engineers. I don't know if you ever do you ever get hazed? Yeah time or two? Yeah, progressing from help desk into like the network

operations, like working on data center. And so we had these two sites and it was going to cost us something insane like one hundred and twenty thousand dollars or something wild to run internet like under the road to our other office. And we didn't want them like using dial up or anything like that to like dial in heart surgery center, Like we need this thing to be online.

And so we did is we installed microwave dishes on the roofs pointed at each other, and we had point to point microwave internet between between these two offices. And they started having outages and they were very intermittent. They're very random, and they're very brief, but like brief enough that it would like stop like an upload or you know, like some sort of scan upload they're scanning back to like our medical record system. And so we thought it was

birds. We thought birds had gotten on the on the things. So my boss is like, like, we we like how we can fix this. I'm gonna throw some of those little bird things on there, like stop them fronts. I don't know what putting those on a microwave dish actually would have done. And he's like, what's confirm that birds? First? I'm like, what are we to ask them? Like? He's like uh no. And I was like, okay, well, what are you gonna do? And he's like come here and he walks out to his truck and I should

you not. He's got one of those umbrellas with a stand on it, a lawn chair, and then he had to walking talkies and I'm like, he's like, you're gonna this is Florida the summer. He's like, you're gonna sit on the roof and when they have an outage, I'm gonna call you on the walkie talkie and you're gonna tell me what you see. Plus So I'm sitting up there for like six hours and all of a sudden,

walkie talkie, hey h surgery centers down what's going on? And I'm looking at I'm looking at the thing and I'm like, dude, there's no birds on this thing. Like I'm looking at like there's there's no birds like I don't know why. I still there's no birds, and so like out my little like fox hound thing, I can still see that there's like signal going through the wire. There's just out and then all of a sudden it's back up, and I'm like he's back up. I'm like, I'm stumped,

Like I'm stump, like this was not in the net plus book. This wasn't in the A plus book, Like I got it, I got an MC. I see this was not on the test right like, And so I'm sitting there for like a while, and maybe like two or three hours later, goes out again, and I'm like, maybe it's a burn on the other side, Like maybe it's a burn on the receiver. That didn't even think about this. I don't know if everybody else is like, it's

a burn on the other receiver, you idiot. And so I look across the street and I don't see the satellite, but I see as a semi truck parked on a restante. Like the other building was a house that was converted into a surgery center, and we were in this essentially this strip mall that had been converted into our data center and like our primary care physician's office.

So this thing's like thirty forty feet tall, and this house is you know however tall houses, and so the beams just hitting the side of this truck probably packed it tons of stuff, and so with longer polls. Yeah, meanwhile, the driver who had frozen pizzas in his truck shows up with fully cooked pizzas at the destination of just like blast and I don't I don't even remember how faster that was with was the one hundred megabit I think back

then for like a network. I don't know what. I don't know what the satellite, a little satellite this thing did. But yeah, he's like I got fourteen cooked pizzas and a brain tumor or anything else, uh helped us. I don't know, Like I kind of want maybe when I retire, I'll just go back and work in it for a year where it's like it's probably a lot less stressful when you don't need to do it to pay

your bills, Like I bet it becomes pretty fun. Then it's just like, okay, like what kind of weird shit do y' all have for me today? Right to see how far you can push the envelope? Oh my gosh, but do you have a good help desk color story. Yeah. So the first help desk job I had was working for a manufacturing company that

had a huge sales department selling their product. And you know, this was this was like nine two thousand era, and so they the salespeople were just coming out of the stage of having their roll of decks where they would pick up the phone and dial and then write down in a notebook what they were doing. So they were trying to get them to you CRM software, and at the same time they had to build computer skills, you know. But I get this call from this sales guy. He's like, hey, I'm

out of disk base. I can't save anything on my computer. And I had the ability to check their disk base and I looked and I'm like, now you've got You've got plenty of disk space. He's like, look, I'm telling you I cannot save anything on my computer. I don't know what your little, your little magic thing down there does, but it's wrong because I'm telling you I cannot save anything on my computer. Like, all right, I'll come up. So I'll go up there, up on the third

floor. I go up there, find him in this sea of cubicles, like, Okay, what's going on? And he's like look, and he shows me his desktop, and on his desktop he had icons for every saved document he had sent to a customer, lined up from left to right, and there was not room to put another icon there. He's like, I can't save it. There's no place to put it. Yeah, he was, he was. He was geographically out of space, That's what he was trying to say. Right, I was writing down what I thought it was,

and that was not it. You guess. I was guessing he had the hard drive you're seeing that, but he was trying to save something to a floppy diss Yeah, it was too big, but it definitely wasn't the grid on Windows ninety eight. Oh my gosh. It is fun. It is fun, but it is also frustrating. So be nice to your support people. Yeah. Absolutely, Like with a grain of salt, understand like who they might have been talking to just before they got on the phone with

you. It's so hard. It's hard, like too when it's just like, oh man, like like making it. I always felt bad when somebody's like I feel dumb, and it's like, I mean, like I don't know I'd feel pretty dumb if I was doing open heart surgery, like I'd fucked out, like like you shouldn't have to know. But uh, oh gosh, that's funny. Cool. Well, Corey, it's been great having you on the show Man. This has been a blast. I have had

a great time. This is super fun cool. Well, I'm happy to have you back on anytime, because I'm sure we've got more stories, especially healthcare stories. We could go down the healthcare path. I worked as as a support operations engineer for a teleradiology company, and there's just so many stories coming out of that that are just too funny. And I think it's been long enough now where like any like anything I could get sued for saying has past, so I could tell those stories now. Oh my gosh, I

have I have one that can't be told on camera. That is right mind boggling. I don't know if, like if you had this experience. But doctors, doctors all think they're God, or at least most of the ones that I met, Yeah, when they're doctors that like own like that. I work with a big practice and the doctors were all like board members and shareholders of the practice, like ooh, that ego is gets big, but we had one where his ego was big and his sense of humor was brutally

messed up. Yeah I'll hear that story once, but not streaming live anymore because it is not it is beyond rated. Are right on, And that's a segue for our new podcast, Things we can't say on the air, So, oh my gosh, listen. No, sincerely, thank you. I've had a blast. I appreciate you coming on the show, and thanks to everyone who hung out for us this entire time. I hope it was helpful, entertaining, and I hope to see you all next week. Yeah,

if it wasn't helpful, I really hope it was entertaining. That's like it was over in life and life. But they're always competing, and yeah, if you could just hit one of them every once in a while, all that's that's winning. If I give somebody a ball, alright, alright, see everyone, Hm