Unraveling Tech Challenges: DevOps Reflections, and Welcoming Warren - DevOps 192 - podcast episode cover

Unraveling Tech Challenges: DevOps Reflections, and Welcoming Warren - DevOps 192

Feb 29, 20241 hr 1 min
--:--
--:--
Download Metacast podcast app
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more

Episode description

The Adventures in DevOps podcast welcomes Warren Parad as the newest host. Together with Will, they dive into a wide range of topics, including technology advancement, disaster recovery, open source, mentorship in the software engineering field, and the intersection of technology and business. They share their experiences and insights on implementing tools, managing risk, and understanding the financial impact of technology investments. Join them as they explore the complexities of the tech industry and the importance of continuous learning and adaptation. Stay tuned for an engaging and informative discussion with our knowledgeable speakers as they share their real-world experiences and valuable insights.

Socials

Transcript

Hey, what's going on everybody? I am your host for today of the Adventures and DevOps podcast, and today is super exciting because if you are a frequent listener of the show, you know that Jonathan and Jillian both have moved on to higher callings, and you know, at this point, I've just got to say that it can't be me. I mean, like, I'm the only hosts that's here that's stood the test of time. But surely it's not me. It must be them and not me. But we'll find out,

because today I have a new co host. I have Warren Parade. You might remember him from episode one forty seven when we were talking about open source software. He's the CTO of Author and he is now the new co host joining me on the podcast. Warren, welcome, Oh thank you. Well. You know, I'm surprised you remember the number, because I for sure did. I'm not gonna lie. I totally looked it up before the

show started. It was actually interesting that you that that one was on open source because I actually just came back from foss Dam UH and Brussels, and there's a lot of parallels there. I felt like it's like the largest open source conference in the world. Yeah, how was that? That seemed like it'd be a cool show to go to. You know, I think it definitely. It definitely cemented a lot of the stereotypes that you can imagine about

both like the open source community. Uh. You know, I go to a lot of conferences because I speak at them, and so like I try to pick the ones that I feel like I can have the most impact on either sharing my experiences are whatever. Have you given my my role? So he said, I'm like, I'm the ct of AUTHORESS Security Application Specialist really, so anything that falls in that space. Actually, it was really surprising when I was there because there wasn't a security track and it's the first year

they even had anything related to APIs. So like my room was like totally packed with people. But it really goes to show you that either it's still a relatively new area and sometimes I forget that for sure. Yeah, I think that's pretty much a rule for our industry as a whole, as it is new, you know, if you consider it too, even like airplanes or trains or boats, Like, we're just getting started here, and I when I talk with a lot of people who are just starting their careers.

You know, they feel like, oh, it's too late, you know, I've missed the boat, and granted there is a lot to learn, but it's it's it's definitely not too late. And one of the things I try to emphasize with that is like when I started my career in the nineties, you know, we just didn't know anything, and we've learned a lot since then, and a lot of that is like foundational knowledge for someone entering

the career. So I kind of consider at this stage in my life one of my roles is to relay that information, you know, and make sure that people just starting their career don't have to spend thirty years learning the things that I've learned over thirty years that I'm going to distill it, digest it, relay it to them, and let them build on top of what I've

learned. Yeah. No, I'm totally with you. I think the numbers rolled up for me and I'm like, oh, wait, Like I remember when I started applying for our positions and seeing high numbers of years of expectations for experience, like even like five or seven. I'm like, I got to get there at some point, probably not. And now, like I'm way past that number, and I'm like, oh, like I'm supposed to

know things. But I mean, I think it's a good reminder though, too, because there's this bias, even especially in our industry, where we think that I think it's experts everywhere believe that the common person knows way more about the subject, and even within software, there's so many different disciplines that I'm often talking to people about things that I think a majority of people know

already and they're just like, wait, don't I don't. I have no idea what you're talking about, Like, all right, that's I totally can't. Maybe it's that surprising. Yeah, No, I think that's been a shift in the decades that I've been doing this too. Like whenever I first started, if you said I don't know about that technology or I don't know how to do that, it was it was a bad thing to say,

and it could be detrimental to say you were expected to know everything. And we've kind of broken that down over the years where it's okay to say I don't know, and and it has to be the vast number of things that we covered, like you're not going to know everything. Yeah, I think coming, like actually acknowledging that imposter syndrome is a thing and that it's okay to have has has been a huge shift that's only just happened recently, like

even in the history of software engineering. I mean, I think it's permeated to other knowledge work related paths that they're they're really as you point out, there's just way too many things that you could ever know. Right, I've too good and stuck because like I keep want to get it into more into AI and I feel like, you know, that's that's its own can of worms. But every time I look, I'm like, Okay, but if I learned something there now, it will just totally be different in five minutes.

So maybe I should I should hold off and just not try to get down that back, right, let it simmer down for a few years, then check it out. Yeah. Right. And that's the other thing too, Like some of these things like if you invest time and effort into them and then a couple of years from now, it's not even a thing,

Like dang, there's three years of my life I'm never getting back. Well, you know, it totally pays off because I feel like my experience in software engineering is filled with the situations that I definitely regret, but they all turn into great stories and experiences. I feel like whereas and that you can

pull forward in some way. I mean I had to be careful about that, like not getting not staying somewhere where I feel like I've learned everything there is to learn and there's nowhere for me to go and it's not necessarily get promoted or move up, just like am I using technology or skills that I'm now trying to focus on or grow? For sure? Yeah? So on that note, how did you actually get started in this field? You know, I don't know if I should go forward or backwards here, so maybe

maybe at least I'll put the goalpost. So I'm the CTO of a company that does authentication and authorization as an API. What does that even mean? I feel like those words are complicated that aren't immediately straightforward, so like user identity and access control specifically for businesses with complicated user roles and resources, multi tenantcy anything in that domain. So it's been a long journey for me to get there. I moved a lot around in a lot of different industries and

locations, so I actually didn't even go to school for software engineering. I went for my undergraduate degree in engineering is electrical and computer and that means I wanted to build Spice satellites, but that actually didn't work. Yeah, I know, that was my you know, it was that or go work for NASA or the equivalent. And interestingly enough, I you know, sort of got an experience of doing that later. But when I graduated, I couldn't

find a good job. I'm by good, I mean like the year before I graduated there was a huge crunch sort of similar to what's happening now in technology, so I wasn't I didn't feel very optimistic about my opportunities when leaving, and so like I didn't get that many and I ended up in healthcare. It in the middle of nowhere in the United States. And let's just say my hobby programming practices of since I was like fourteen years old, I

think that's about when I started. We're more sophisticated than what this company had going as far as like software development processes and source control, which they didn't have at all. So that was, you know, quite an initial shock for me, and I really didn't want to stay there very long. I think what was interesting about that job too, is that I didn't have experience in hard software engineering skills, but I did a lot of classes that I

think are still valuable for me today. So like, I don't care if someone went to a university when I'm hiring them or not, but there is some background that's super useful, like algorithms and structures and understanding what that is, like if I say hash map or dictionary that sort of comes with some expectations. I don't care if you know how virtual page file system works, because I still can't figure that out for the life of me. Yeah.

So this again actually wasn't even software. I was doing technical services, which was pretty much doing custom software development for the customers of the company, but not building the product up. And that was actually really working with customers directly,

our users to help them understand how the technical product worked. Super complicated, lots of configurability, so much so that customers could put in their own code into the system, which is sort of ridiculous that you would go to production and make changes there directly in the operating system level, and like that scared me that anyone would do that, but that that was the thing.

So that also wasn't wasn't pure software, and that was great because I feel like I got experience with Okay, what we're doing has an impact to a user somewhere, like there has to be a reason for doing this. Yeah. I think healthcare is one of those interesting places to work in technology because

you have like all of these different constraints. You know, you have just like the whole world of medicine is huge, but then you have it's implemented differently in different regions, and then you have like the legal aspect of it

with hipper requirements and stuff. One of the jobs I had was it was an This was before the term SRE was around, but it was kind of an SR type role where we worked directly with the staff and trauma centers and so whenever they would call us and say, hey, this thing's broken, right, they had somebody in the emergency room on the table that they couldn't

give medical care to because our stuff was broken. And so you were talking with you know, doctors and nurses, and there were there were many occasions where you're just tell them, hey, look, just put me on speakerphone, go do what you gotta do, and I'll shout if I need anything, and I wish I wish I sort of had that experience, because that seems like one of those that you know, falls in the category of I'm glad I did it in my past, but it's not my day job anymore,

right. I Actually I actually worked in claims and like billing primarily, so I got to deal with all the problems of configuration in this system and also how to deal with those contracts with third party integrations to insurance companies or clearinghouses. Yeah, insurance companies are just a joy to work with at every

level, right. I mean, this is like after a standards body came together and said we need things like hl SAT then to perfectly identify programmatically how to communicate, and then the hospital organizations would go out and sign contracts with these insurance companies that violate the data that's allowed to be put in the programmatic communication and the interface and their APIs or EEDI interfaces depending on where you're from,

and like that was really a struggle to communicate where like, hey, you know, our software doesn't do this because it's against the spec like this field, it takes a Booleian true or false, and you can't put a three in there, and part of it was to protect the patients, like what information should be allowed to be sent to the insurance company that wasn't relevant

for paying out whatever diagnostic or surgery was assigned in that claim. So I did remember one time, and this is where I'm glad, I'm not I never was in the sor repath directly that one of the data centers that one of the hospitals that I was supporting had an incident and their like main systems were all offline like totally, which you know, same to do your case, and what they had us do was remote into the systems, not using

SSH that would be cool, but using some other technologies that I don't want to name, and like three different levels, like we had to first we got the beeper go off, you had to get on call giant phone room where then you got a sign credentials in order of access. Then you could get in and then SSH to another system and then to another system to even get into production to run a script basically to verify that the database was integrity

was still you know, okay. I didn't like that. I think one of the big takeaways for me from working at that job was just putting things into perspective, you know, especially since I still deal with production systems a lot in incident management. You know, after that, my approach is, oh, we've got an outage, Okay, how many people are going to die? Zero? Okay, Well, let's just simmer down a notch here then, and let's just make conscious focused decisions here because everyone's going to be

okay. Yeah, No, I totally get that. I think this is one of the most frequent advice that I end up giving inexperienced like new engineers the field is really just take a look at that risk profile here of what's going on, Like I need help immediately. I'm like, do you? I mean, your production deployment isn't working, but the site is working at this moment, and it also doesn't serve critical usage, and also you don't have an ongoing marketing campaign and maybe you have five users, right. I

think I saw an XKCD on this. I don't remember if it's that or someone else, and there was like, is this is like posts like my SQL or postcress or something. Is my database configuration will work for having one hundred million concurrent users and the responses how many current users do you have oh we have five? Oh yeah then yes, yeah, right, I mean, because there's a lot of truth to that. It's just like you you

don't need that scale at all. I was actually talking to someone today and they were doing they were almost exact same question, but they're like, we don't want to use an identity provider that's proven hardened or whatever. And I'm like that's okay. They're like, oh yeah, it doesn't scale for us. And I'm like, how many use it? Like ten thousand users? And I'm like, okay, you know that's just like such a small number, right, not even here. I'll let you borrow my raspberry pie that

work, I know. Right. The thing that scared me is the difference between it being like a commercial application for personal use and when that goes to a business. Because as soon as you bring in business as if people are asking these questions, I start to get concerned, like, Okay, what is the data privacy that you're thinking about? You know, what is your

data redundancy or disaster recovery situation? Because if you're worried about spending ten bucks a month on running your app, I feel like there's a promise here to your customers. That's more than just that could go down and having a backup strategy just it's like number one for me. If you're in the business space over you know, who cares if you lose some music files from your Spotify

account? Right? And I think that ties into something that is very specific to devlops but also applies to other areas of technology too, is like it's not always just about the technology, Like you have to understand the business as well and understand like what your promise to the customers is and how it impacts their life and even like in in some cases, you know, you have to understand the finance aspect of it too, because we've build all of this

infrastructure and now the finance team doesn't really know what to ask. But if you go to them and say, hey, some of this infrastructure is op X and some of it is CAPEX, and if you work with us categorize it correctly, it can result in a huge amount of tax savings for the company. And so you get into all of this non technical aspects of a

technical business. I love that. I love that you brought that up because I think that understanding even the difference between those two is such a foreign concept too. And sure, in general, I think we get lots of customers that are like, oh, it costs ten thousand bucks a month for your service. That's too much. I'm like, let me explain to you,

you know, the savings here. Also, you probably have capital expenditures that you're currently paying in and saving those resources that are depreciating over time, versus paying a SaaS provider to you know, deliver some of that. And if I look at their use and it's like peanuts, like a couple bucks a month, it's like you're literally wasting more time trying to make a decision on which statas provider to use for something than you are to pick a random one

and paying the cost to transition later. It's really sad, honestly how often I have that conversation. Yeah, you have to you have to know. And I think this just comes from experience of knowing when to dig into the weeds and when to take a step back and see are we even in the right part of the forest here. Yeah, No, it's interesting because actually

you know to Can you continue on my career past story. I worked at a large global manufacturing company that wrote their own software and they actually did chargebacks really well and expose the cloud costs eventually to the individual team so that they

could take ownership over how much they were spending on some things. And you know, I think that's a great a great first level, like giving that exposure rather than locking it behind another team or even a different part of the organization like one quote unquote finance, because you know, that was great to see, Oh yeah, you know, if we' spin up this infrastructure,

it's going to cost us two thousand dollars more a month. But then it's great to also hear that other teams are like, well, well, whoops, we just spent two hundred thousand on the others sasas provider. I think I think I read it it was like in the ten day of like Coinbase or something that they had spend sixty five million dollars last year or two years ago on data dog or something like that. That's actually really easy to do

with data Dog. I mean that's always been my concern. I mean, they're not the product for us, given where like the sort of data that we want to collect and the sensitivity of that data. But it's also really amazing that there aren't any product that really fit the space for us. I wish someone would come to us and be like, you know for metric usage, you know, here's a perfect thing, but you know, we still

haven't found that. Actually, So we actually because in my full time job, I'm a staff engineer and the DevOps team for Polygon, and we actually just rolled out a product called cloud zero that has been super cool. It does, it collects the it connects to whatever billing APIs for different providers that you use, pulls that in and then gives you insights on it, and it's really really insightful. Like the very first day, it's like, hey, here's a bunch of net gateways. You're using an AWS, I don't

have any traffic. You can save you know, five thousand dollars a month just by turning these off or and it's like daily emails. It's like, hey, over the last twenty four hours, the spending on this GCP product or GCP project went up one hundred and twenty five percent. You're like, wow, that's a jump. And so that's that's been really cool just to manage those those kinds of things, and it's been very I'm not trying to

plug Cloud zero or turns into or anything. We just had good experience with it because they've just they've brought up some stuff that we would have not seen, you know, very easily, and the integration was that was super easy. I mean, I think I think that's the truth is like those sort

of tools have like significantly evolved recently. I think at the time, which was almost eight years ago now, we were using like I think it's called cloud and area or something like that, and it was it was not it was not particularly good at the time, but yeah, being able to see those trends in not an amateurized cost way was was helpful and actually driving alerts and whatnot, which at the time AWS didn't really support. AWS doesn't really

have motivation to support it. Well, you know, I think that's sort of the sad story of SaaS products is I mean really everything in general, that once you find your core customer, if they're not going to pay marginally more money for your same product, even if you make it better in some way, and that's just like the law, like I pretend that doesn't exist, Like I want to happily live the rest of my life believing that if

you make it better, people will appreciate that there's a brand impact for us. Like a very common conversation I have with my CEO is about how well the quality of whatever we're doing before we start going down another path, because I think it does really reflect well users bringing in more customers, et cetera, rather than secretly complaining about it behind closed doors. Yeah, I would agree with that because I you know, you obviously, with a good internet

based marketing campaig, you can reach a huge audience. But there's something to be said for just delivering a great product and having satisfied customers, and then when they go to their next job, they're like, Hey, I used this before and really liked it, And I think there's it may not have as big of an impact as a funny marketing campaign or whatever, but I

just think there's something professional about that. Well, it's interesting you say that, because I'm not convinced that in the business space that even viral marketing campaigns have any real value there for sure, if we're talking about personal products, like right, yeah, definitely, But there's a whole There was a whole initiative by the search providers AD marketplaces quite a few years ago that tried to actually figure out whether or not the traffic that was being sent to your website

or app actually converted for you. Right, obviously that's a big importance. But if it didn't convert, you almost got dinged because you basically lied about your advertisement, right, you know, traffic when to you user didn't get what they wanted. So I think it's I think, you know, originally the original version of this is like sex cells, and I think the truth is, well, no, sex just makes people click the link. But after that then they then they drop off because you know, it's not actually

what they wanted. And so at least for our content that we put in, like our knowledge base or our blog, it's like it's got to be super relevant because while there's lots of things that we can say that are really interesting from a technology engineering blog sort of way, those don't really help get customers in the door brand. I mean, Brandon Winness is one thing, right, you know, you must your brand all you want, but that's not going to really result in direct sales, but it will result in this

concept known as oh did I forget the name. It's like positive reinforcement. Yeah, yeah, sort of along the lines of like someone has to hear your name five times before they ever recognize your company name. Oh, that's a high number. I didn't want to hear that. Don't don't quote me on that number. There's a rule that marketing teams use that people have to see your ad or whatever a certain number of times before they start to recognize

your company name. But I don't remember if the number is actually five or what the value is. I mean, I'm sure the unfortunate truth is it's more than you'd like it to pay, right, right, that's the takeaway there for sure. So looking back over your career, what are some of the things that you some of the career development decisions you made that when you look back, you're like, ah, that was just pure brilliance. Well, I don't know I have an answer to that's question, or like,

Okay, that wasn't completely dumb. Okay, let's set the bar a little bit lower there. Yeah. I think I made a couple of realizations while I was working on different things. So one of them was like, the first DevOps related thing I had ever done, which was not called DevOps at the time, was helping the organization I was in to replace the need of a different organization, which was called Release Engineering, and the third organization that

wanted to help us do that told us the solution was Puppet. And if you don't know what Puppet is, it's a convention based Ruby set proprietary packages really, which is just the most ridiculous thing that you install agents on all of your machines to deploy and upgrade the software. The thing is is that you pretty much to do the equivalent of writing your own terrorform provider for your own software in order to deploy it. There wasn't anything common, really,

and what was common didn't really work out of the box. And like you will, I was the only one left on this project by the end, from three teams that hadn't been swapped out in some way, like twelve people left it over the time I was trying to get this done. But the point that I bring this up is that repeatedly it was said, usually by me, oh, if we wanted to do it that way, maybe we shouldn't be using Puppet. And at the time I was literally I was so

under leveled. I was literally the lowest level that an engineer could have at this company at the time, which its own sort of travesty, which should teach you something that your ability doesn't really mean anything when it comes to the level that you put decide if I had been a higher level, I would have really after the fact that saying those words to myself or even out loud,

is an indication that you should do something about it. And I mean, in this case, if I left the company, But that's something I created. This rule. I mean, I call it the rule of three really, but like if I hear the same problem in my vicinity three times, then I know that it's on me to solve it. That no one else is going to come up and just magically resolve this issue. And I mean it's not like a hard and fast rule or anything like that, but

realistically, no one else may be privy to it as much. Right, Like I may have three different sources that are all coming to me or talking about it in my vicinity. So I usually watch out for those things because they help a lot. So like this was one I should have realized, you know, after the tenth time I said, maybe we shouldn't be us

puppet like, we should have picked a different one. The sad truth is I think that company still has my like the system now I had helped build there to integrate with Puppet being run by like out of the three organizations that

were set up to manage this infrastructure. Basically there's like a team of two or three people now that are handling like all three components, which is just ridiculous really when you think about it, because this is the wrong technology no matter how you look at it. And they don't have the skills to build to even manage the rappers or abstractions that were built to manage this, and they don't have the experience or understanding or knowledge to do it either, So

it's really really unfair. But the biggest lesson there, I think is don't use Windows. Yeah, I mean, so this is a manufacturing company. At the time, there was four manufacturing plants all over the world, and they had a monolith. There was something like one hundred and twenty Windows Services, which amounts to like it was like eleven hundred c sharp projects that were

being built and managed and deployed in a monolithic fashion. So like they had a real need for strategy strategic change of how they were doing deployments, which and I think this is a good story if you wanted to make a new service, which like this is still in the days of say, like the term micro service didn't even exist yet. You had to contact this second organization and say, I want a new micro service. Except you could only do

that. You needed a three week lead time for them to create the infrastructure for your service. And given when you had to do that and the lead time it would take for you to actually release your new thing, you could only do it once every five weeks, and it was one day of the week that you could do this, so literally every once every five weeks. I'm like, oh, there's that. Like I couldn't believe this when I

figured it out. It was like they have a process, and we have a process and together that eliminates every possible day with it a one plus period except for this one specific day because we had code freeze and you had and you had to introduce the service and then have a handoff and then other nonsense. I mean, that's why we were going to puppet in the first place, which was supposed to be the revolution never really made it to be right,

it was going to automate everything for you. Oh yeah, for sure. The claim to fame there, like the interface I provided for other engineers, which I thought was absolutely fantastic. Was a single file that could merge in lots of different version changes from individual parts of the monoliths changing at any time, And so each line of the file was a because the new organization said that ivy was the way to go, which is a Java resolution sort

of package. I mean, it's not even a package manager. It's a JAR dependency manager, which is a sort of own atrocious thing. This was a HTML file well that had service and version as h al attributes. And to make it worse, we get every other line was a comment that said, do not remove this comment because because if you're not aware of this when you make a change and get, if there is no change on the line

above or below, then you can merge very easily. But if there is a change on the preceding or following line, then there's going to be a merged complex so unnecessary line. So yeah, I mean there was a comment between each one of these that said, you know, do not remove this. But I thought this was a great interface, and for sure, you know, I would never never do it that way ever again, but it

was the right thing at the time, for sure. Yeah, And that's one of those things where, like you you know, it's not the permanent solution, but it's a foundational step in building a permanent solution. I mean, I wish someone had said those wise words to me at that moment, because it is certainly the belief that I had that this would be the only thing that was ever necessary for the rest of the life of this company. I mean, I guess the fact is still running is some truth in there,

no, But I think that is a good point worth elaborating. Is like everything we do is, you know, without trying to sound like philosophical, everything we do is temporary, but just meaning that technology continues to grow every day, and so whatever we do today is really just a stepping stone

to whatever is beyond that. And if we just like do like a really high level history lesson you know, we had servers in closets at the office, and then we had data centers or we would go wrap our servers, and then we figured out how to run virtual machines on the servers, and then we got you know, cloud providers like AWS where you could run virtual machines, and then containers and and so like all of those were just building on top of the lessons learned from the previous decisions. I mean, I

think that's an interesting point. I think it was like, not too long ago, I was reading a paper that suggested the persistent media storage that we had over time was becoming more ephemeral, and like the replacement speed of things like the time between SETE cassettes and I don't know what else CDs and then VHS tapes and DVDs to Blu ray to you know, USB sticks slash drive

was getting shorter over time. Ah, And it's interesting like our is our technology that we're building now more ephemeral, transient than what we had in the past. And I think what really missed out in this article was thinking about the lack of stability of volatility of the previous generation. I mean, the I'm gonna get this wrong. The US launched deep space probe that contains the gold DVDs with is this Voyager? I know, I don't want to I

could believe it's called Voyager, you know, I am not. I am not the exit export name recognition here. But yeah, right, like we put it on gold discs, right, would we still like, I mean, I have no idea is that better? Than you know, some other less ephemeral flash system that we have now, Like, would we still use that same strategy? I mean probably not. I'm sure someone dream up it's like some sort of antimony composite or otherwise, but it's still like is it

metal with like physical embeddings in it? I don't know, So I thought this was interesting. I would just product tape a USB stick onto the satellite. I mean, I think the realistic is like you just put fifty USB sticks in the satellite, like its all as like one of them needs to survive. But then you like you figure out like Hoffman coding and error correction and so like it's okay if like half of each one of them survives because you can put the data back together. Just load them all with a multi

petabyte ZIP file bomb just for laughs. Some aliens find it and it just blows up their computers. Yeah. Well, I mean it's like running commands is always weird, I guess, right, because you have to know what the language is in order to even execute that thing. Otherwise it's just you know, nonsensical. But I think that that's really that's really the rick roll, right, like just just put literal nonsense on the disc and set that

up, you know, a society. Yeah, I remember this an outage I was on many years ago, and it was storage related where a vast majority of our production systems just had no data. And so we're on this call troubleshooting it for hours, and I get the storage team on the call, you know, and they're like, yeah, well we had some disc failures and so we swapped out the disc and like, okay, well that seems reasonable, and how's the rebuilding going. Yeah, the rebuilding seems to

be stuck. Okay, And we talked this out over a course of a couple of hours and found out that nearly every day they would come in and a disc would fail, and so they would swap out the disc. But then after a week or so, in the twenty four hours between the failures, that wasn't enough time to rebuild the disc. But then they would come in and swap out another disc. So after a week or so they'd swapped out enough discs, we're in this storage array where none of the dis had

enough information to do it, so they lost everything. Wow. I mean I wow, that's you know, and then you go back to okay, well, let's let's restore from tape. When's the last time you've verified your tape backups? Yeah, and you get crickets like all right, well we're going. I mean it's like this is like one on one running fire drills on your that's the recovery process. Like, yeah, forget verification, Like did you even like you literally need to load the date in bload the data

and be like yeah, we're switching over to it for real. Otherwise you might as well not happen. Yeah, And I think that's in my experience, that's been a hard thing to prioritize. You know, you just essentially have to say, look, this is what we're doing. You may not like it, but there will be a day in our future where you are happy that we spent this time because you learned so much in trying to do that and you recognize all of the things that you thought you had covered that

you don't have covered. Yeah, no, for sure. I mean I think that's one of the areas that's really undersold by the cloud. Like so often I used to do before my current thing, I was doing some fractional engineering CTO advising consulting, and it's too expensive for us to move to the cloud because we looked at our current infrastructure that was on some on prem data center or delegated and we just calculated the exact number that it would be if

we tried to run those exact things in AWS. And the ridiculous part is like they're not accounting for the actual value add of not having your tape backups, you know, right array failure issues, because that is like if you care about that, and you probably should actually care about that, it's non trivial to have that working correctly for sure. Yeah, that was whenever I got out of high school, I joined the Navy and became a nuclear engineer.

And one of the things that they did there was they just really drilled home the whole disaster recovery thing. You know, you would be operating the power plant at two am, and you know, some master chief who was up and couldn't sleep, he'd go walk down into the power plant and just turned something off just to watch everyone respond, you know, and it really reinforced the disaster recovery drills so that whenever something did happen unintentionally, just those

skills we were just muscle memory at that point. Yeah, that really Also it seems like an exercise and root cause analysis, right, because which is not practiced enough realistically. I mean, there's a lot of I don't know. I think there's a lot of companies out there that talk about post mortems and one on, but having been in quite a few in my day, I don't usually see them run in a way that actually has meaningful outcome there,

like actually resulting in better processes or practices in general. I think the number one thing that comes out as well, we should have better tests,

right, we should have more tests. That's the solution, more tests, And that's usually what happened, more tests, usually by adding some sort of rule to your CICD platform that ensured that your code coverage was at above eighty percent, which is always just great whenever you delete code and it goes down because of course you deleted code with tests on it, because that's usually the

code that's tested the most worthless not necessary. So and I'm just like, I don't like I have stopped working contributing to certain open source projects when that happens. I'm just like, I don't know what to do here, Like I don't know enough about this service or SDK library to add tests somewhere else because obviously they're not there for a reason. It was deemed too difficult to add them. And now you're asking someone who doesn't know what's going on here

to fix a bug by deleting code and you can't. You can't do it, yeah, for sure. And every test carries an equal amount of technical debt with it because the test has to be it has to be actively maintained and evaluate like is this test still testing the right thing? And when it fails, did it fail for the right reasons? Yeah? Yeah, no, for sure, for sure. And I think there's a whole, this whole h extra hours of uh content here that may not be directly related to

DevOps that you go on with. I mean I worked with a lot of originally QA but eventually QUI and just selfware engineering teams trying to turn it all into the DevOps mindset, because that was a new term, I guess during my tenure at different companies, and not one that was always so practiced readily,

for sure. So what were some of the things in your career If for someone who's just getting started or part way through their career, what were some of the things that you did that didn't seem to have any value or were perhaps negative to your career, Oh for sure. The one, My favorite one is telling other people how their solution won't work. Yeah, that's rare, my favorite. I thought, you know, I was doing the right thing early on. Hey there's this thing that doesn't work. I mean

I not so long ago. I actually did a presentation about how the human psychology doesn't respond togative to terrence or even really any sort of constructive feedback very effectively. I mean, there's a whole psychological safety aspect and desire to change

of which most people you know don't fit there. Most people care more about being told that they're right or they're doing a good job, which says a lot for positive reinforcement that you can get people to do the right thing, to be motivated even if you congratulate them, if you tell them how they're doing good job, give them rewards. Just don't put this the carrot in front of them too often or explicitly, because it will. It can derail

where people get personal satisfaction and motivation from. But it helps way more than anything else. Like, I think there's this common misconception that's widely acknowledged that performance improvement plans work in some way. I mean, it's a great example, right, you know, hey, we're going to fire you if you don't get your act to get other never really fix anyone's career specistically, it did help, right, you know, those were the words that someone needed

to hear. Yeah, yeah, but you know that was a big one. I think another one for me, and I was lucky that I had a couple of mentors that really shared this with me specifically, is that it doesn't matter how good of a job you do. Realistically, no one cares. They care about the outcomes that are important for them, and so figuring out what actually is important for them. And it's not like some them,

which is some vacuous, morphous blob somewhere. It's you know, put points to your finger at whoever is responsible for giving you a promotion or who I mean, realistically, whoever cares about your future development, which you should know who that is and how they're going to help you get to the next level, and ask what's important for them today, right now in the future. And that's the answer, Like everything else, totally irrelevant. I mean,

do it makes you happy? Right? Like you shouldn't go to your job and be unhappy, so you know, find a way to work that in, but don't for one moment believe that doing a good job. And this was like some lie that I had told myself, and I remember thinking it so many times. I know there are these things called politics. I'm not going to get involved. All I care about like I will get promoted by doing the best work I possibly can. And I just stop using the word

politics because it doesn't like it doesn't really land. It didn't land for me, It doesn't I don't think it lands for other people. Really, just think about what you want and then draw the steps to how to make that happen, which are usually convinced someone else that you did a good job.

And that just involves a conversation for sure. Yeah. Yeah, there was a quote and I can't remember it exactly, but someone said, just because you choose not to play in politics doesn't mean it's not a political game. H Yeah. I mean I think I think for me, there was the like this way, you still twist it for yourself, where like, okay, there is still a game, but I'm going to try to get around it in some way. And I think I think sometimes assigning those labels really

hurts. Like another one I think really wasn't great for me in my career. It was people calling I mean, I don't know if you can believe this, but apparently early in my career I lacked what are called soft skills, you know. I think that goes along with telling people that they're wrong, you know their solution won't work, and then also be like I'm very

to the point and very blatant about it. So, you know, a couple of those things together, and sometimes people's feelings got hurt, which of course was never my intention, which is totally different than all those people that do it with this sort of negative or malicious connotation to it, like it wasn't about you, it was about like, literally, this solution doesn't work.

I think for me, what really helped to stop calling them soft skills, stop calling it Paul politics, and focus on what it really means, which is you can work way more collaboratively with other people if they like working with you, right, And when you phrase it like that, I feel like it much more helps people who maybe it were in my position or similar positions who maybe don't go along by default or different cultures engage with each other

to take a step forward than it would if your manager just says, hey, you know you're missing this manager buzzword that doesn't necessarily connect. Yeah,

for sure. I think you know you mentioned having mentors. I think that's been really I've really taken a completely different view of that over the last few years, because you know, like I grew up in the middle of Texas in the seventies and eighties and can honestly say, I don't know that I ever heard the word mentor until I was in my twenty and probably never used it in a sentence till I was in my thirties. But now you know,

it's a very common word. And I used to think, well, I guess I didn't have any mentors, But then I the more I thought about it, the more I realized that I did. I just didn't know what the word was for them, and it wasn't a negotiated arrangement. And I think that's still true. You know, like you can have mentors without having a formal conversation. I have this contract here, would you like to be my mentor? It's very low offer and low commitments, easy out.

You know, it's not that type of thing. Like a mentor can be anyone who who says anything to you, whether it's helpful or hurtful, and it doesn't have to be like on your terms. And what I mean by that is like, you know, people have their way of communicating, you know, Like I said, grow up in Texas and then I went to the Navy. And this was back in a time whenever being in the Navy, your performance was judged on how many bar fights you got into without going

to jail. So it was, you know, it was a different environment. So whenever, like your master chief told you, hey, you ever pulled that shit again, I'm gonna kick your ass. You know, that was a form of mentoring. You know, it wasn't it wasn't a nurturing relationship, but it was a mentoring relationship. And so I think that was one of the things I've learned over the years is that mentoring comes in all forms and it's not a predefined, negotiated relationship. Yeah, no, that's

I think it's even more effective when it's not. Like I hear of companies trying to start some sort of mentorship program, and it's usually like my company is starting a mentorship program? How do I make this be effective? And

I'm like, well, why are they starting it? Like you're getting in on the wrong foot almost, And you say that, and I'm thinking back to my own career and I feel like every job I was in, there was always someone who I gravitated towards and because maybe they were more experienced, but it was never like, hey, can you be my mentor? It was me asking them questions when I when something came up, trying to get

additional information or understand how they did things. I think that that definitely helped makes like going to the right people potentially that I think I would definitely classify as as mentorship. So you asked about things to do early on in your career, you know, find people that can teach you something, and they

will absolutely teach you those things. I hate to quote Steve Jobs, but he has an example where he called out and asked an executive for feedback to help them and they're like, yeah, sure, I how can I do that? I think it really is that people will I think Adam Grant says the majority of people are matters that most people will reciprocate un asked. Really, if they think there's a good engagement for them or could be a good

relationship or whatever. It doesn't have to be like something static like put a label on it, like it's not necessarily required. Yeah for sure, for sure. All right, so there you go, Welcome to the podcast. Yeah, I mean, you know, even here I I I before this, I was asking you, you know, personal questions about how you how you run this, you know, as a good as a good example, you know, come ready, how do I do this effectively? Just because you know, I want to do the best job I can and I'm not

the super expert in the area. Obviously I pull skills from what I have in the past, but looking for people that have even just slightly more experience than I do. Yeah for sure. So what do you think should we move on the picks? Yeah, I'm trying to. I'm trying to think if I have any you know, most most important thing that I you know, ever ever wanted to share here. And I feel like I have so

many things, but probably not super relevant at this point. So yeah, let's let's do the That's the great thing about picks is it doesn't have to be relevant at all. Just be like, oh man, this is so cool, you know. And so we were talking about movies before we started recording, and during that conversation I learned that, Okay, you obviously are a movie guy. You have your own IMDb account where you save and rank

your movie, so I know you've got something there. He wants he wants to hear my favorite movie of all times, all time favorites, you know. And that's the thing where I'm looking at this list, and I have twenty two movies rated at ten stars, which is the maximum, and so I feel like even picking one of these is just such such a challenge. Realistically, It's like picking a favorite kid. You can do it as long as the other kids as you have, don't hear you. I mean,

you know, and I like them. I think they liked them all for different reasons. That's the interesting thing. Like I have ones that are film noir, which I as a genre I really like, But then I have ones that just for whatever reason I liked and had some emotional component to it. For instance, you know what this one I think I saw recently, So I'll say this. I think you know, it's not my top one, but you know, this is the one I'm going to pull out this

contact. It's if you haven't seen it, it's with Jodie Foster and I think they're the sad part. It's about about making contact with aliens. Maybe she has sort of a moment in building a machine. It's about the SETI project where they discover, of course, signal from outer space and t USB drives to the satellite to contact. No, actually they the communication around our space was a machine in ructions for building a machine which they could build.

And of course, like you know, she spent her whole life doing this, you know, through adversity, and she doesn't get to be the one to sit in the chair to go contact the aliens. Initially, I feel like the thing that really hit me is there is it's such sort of like a I don't even know it was intentional, but such a parallel to some of the struggles of diversity in the tech industry, because I feel like she was the perfect pick in every single one of these situations and kept getting gone

over for whatever reason. And like the Catharsis, at the end of the movie, she actually does get to meet the aliens and then no one believes her, which is just like, I mean, I like space movies, and I like that one had me just I felt so bad, like hearing that, like just seeing that story, which of course is not real as far as I know that, Like, I'm sure there are people out there who said that, you know, they've they've met aliens, right, Yeah,

there's been a few, but none of the interviews I've seen have the credibility of Jodie Foster just say yeah, what's your what's what's your peck? So my pick for the week is going to be for the platform Con conference. And we had the guy who created it, Luca. He was on this show. I don't remember that episode number, but he was on last year talking about platform engineering. But platform Con is a virtual conference that's five days long, coming up here in a few months. But it's it's really

well done. And the reason I'm picking it is because at the end of the conference there's going to be a live Q and a session where I'm going to interview some of the keynote speakers at the end of the conference. So I'm looking forward to that, looking forward to to talk with those people and seeing if I can uncover some things that didn't get brought up in the conference,

and that seems pretty awesome. Yeah, Yeah, it's a cool it's a cool conference, and the best part is it's virtual, so you can just throw it on the background while you're doing whatever it is you're doing and you don't have to try to argue with your boss to get you know, to get approval to go to it. But check it out at platform con dot com. And you've got a few months to get into to get signed up for that, so plenty of time I start. I'm going to put

it on my list because I'm sure there's something there I'm interested in. Yeah, they cover a pretty good range of topics, you know, not just obviously not just DevOps related, but platform related and platform really is I'm like in the word platform because I think it's more representative of all of the things that we cover versus DevOps. Yeah, for sure, So that's going to

be my pick. So for anyone who's listening to the show and wants to reach out to you about it, either about authors or just career or just to say hey, when can you get that will do fired so that we can go on with a decent podcast? How can they contact you? I think for sure? I accept LinkedIn connections if you start a conversation, I will I will I kick people out of my connections if they don't talk to

me on LinkedIn. I think knowing who your network is that is valuable to you is valuable, and it's difficult to go through your rolodex if you don't know who you can actually talk to. But I think fundamentally the best place is on discord. I'm in a bunch of different Discord communities. I think we can get the link in my host profile or something like that for sure. Yeah, And that's definitely going to be the best way. And I happily talk about whatever you want to talk about. Their cool right on.

My wife spends a lot of time on Facebook. Not a lot of time, but she's on Facebook. But she has a similar rule where she only allows fifty people as her Facebook friends. So if someone else wants to be her friend on Facebook and their number fifty one, she's like, I got to kick someone else out before I can be friends with you because fifty is the limit. So there's two really good points about that. First one,

I'm doing the exact same thing on Facebook. I mean, if we think about Dunbar's number, which is not a fixed number for the record, it's just not possible to have that many close connections, so realistically, you should keep people in the loop that are relevant for you. It's way more effective and you'll be happier when you are focusing on your connections that matter. On LinkedIn, it's slightly different. I don't put a fixed number there because there

is a advertisement effect for my brand. It's sort of like Twitter. The more people that hear me, the better. But I'm not on Facebook and I almost never use it other than I you know, I do need to talk to one of these really close people, and I don't really use Facebook at all besides that. Yeah, agreed. Cool, All right, Well, Warren, I'm excited to have you here and I'm looking forward to many many episodes with you, and I think this is going to be fun.

Yeah, me too, I'm I'm excited right on. Cool. Well, thanks for listening to everyone, and we will see y'all next week with another episode. And I'll stop the podcast as soon as I find my mouse there we go, all right, see everyone. Bye.

Transcript source: Provided by creator in RSS feed: download file
For the best experience, listen in Metacast app for iOS or Android