Achieving Engineering Excellence: Training, Innovation, and the Cloud - DevOps 213

to take that seriously and removed it. And now you can only get endorsed for authorized roles, but no professional Yeah, right, like professional chocolate taster.

is what comes daily into a software engineer's job. I mean, you're not nestily getting tasked of, right these three functions. You gain tasks of solving a specific problem and then it comes in and sometimes the problem is solved by writing an email and not writing any code. So so there is this sense of I want to do something good for the customer and for the business or ideally the mix of both. And yeah, and I think it's a different drive what you have and what type of

skills at what you learn. And I think it goes back to university projects where everyone does it in some form or shape, and you know, motivation there and just trying to be trying to make it the best what you can at that point. But this older naivity what you have, I don't know, fifteen year old in school or twenty five year old in university. And so.

you have another hour of work for accounting and bookkeeping and all of the stuff with running your business. And then there's an additional hour on top of that where you need to do sales and marketing and prospecting and finding your next gig. So even though that hourly rate looks big, you have to divide it by three to realistically put it into.

not making anything for that particular customer. So I can imagine there's a lot that goes into that for a very small startup, especially one in the chocolate tearing space.

optimize for that. But like, yeah, you definitely learn also back to those skill sets and this drive too. I need to justify that hour, right, And I think that's the beauty of external people. They show up, they need

to justify every single hour. And if external people are used the right way, independent on whether they come from a big company type of consultancy or as a one person is they don't have the ow ahead of a company, right, And that's that's I think important as you collaborate with those external people and making sure this hour needs to

value differently than an internal one. The internal one you train that person to be excited in five years to grow internally versus for the other one like, this is the task in a week you on, this is how I can can actually get the value out of it, and also, hey, I paid you that, but you also delivered.

I think that's really interesting dynamics, but often an oversight for those who are on the other hand, so not the freelancer, but actually receiving the freelance work and actually make that, make that a thing, and make make it work and make it valuable.

but it's just whatever needs to get done. But that said, let's let's focus more on the excellence part of the engineering part and probably start with what is excellent, right, there's probably variety of definitions when you look up on the dictionaries and like it's wake, but some of those sentence has definitely resonated to see you it's something like greatness or being the very best, and you know there's millions of soft engineers in the world. Not everyone can

be the very best. Not everyone can be you know, these star singers or star sports and athletes and so on. But nevertheless, you know, look, look look at other disciplines like sports. You might not become, you know, the next world champion of whatever sports, but you can look at them, you learn from them, and you kind of strive to do better even if you know you're never getting there.

You know, you might have your lab times on money and then you just might want to be ten seconds faster in half a year and you're proud of that and you're striving for this incremental improvement. So anyway, so excellence, I think it obviously depends on where you're coming from, but generally it is about the relentless drive to become to envision that greatness and embrace it and make iterations

towards that. And I think it's a little bit harder to find what does this mean in a job that's not so you know, you don't get so many medals by being the best. You don't get you know, you're not this key champion who is the fastest. You're not the golf of who uses the least amount of things. You're not now in the you know, Couper America and the European World CHAMPIONIP, there's just one team winning it.

But nevertheless, those who are already out of the race, they they're part of the excellent strive as well, So let's not forget that. So it's definitely very inclusive of a lot of people here. But nevertheless, I think all of those who are on that playing field, they are they're just like doing one more thing than anyone else. So anyway, so how does that translate a little bit into engineering? And I think one of the interesting talks probably that that many might know is Lenis Tobo's defining

good taste software. And he did that in a TED talk something like ten years ago and showed like, hey, this is how computer science teaches you to remove an item of a single lecklist or something like that, and it had this if condition in it. He must not lessly water by what he is taught by computer science,

But why these if conditions. It's just like there might be a bug in this thing, and there's another if condition added, and it just adds this complexity versus really trying three layers deep of what's the thing, how can you solve it? And how can you solve it in a way that is just like, that's the thing, what

it does. There's no discussions about it. There's basically no way of an alternative pass if the first one and the last one or the middle one and so on, Like all of a sudden, you get into no, that's the thing, no matter which item in the in this list it is. And I think that one is an interesting one in writing code and and I think now these days is a lot of that is a lot of us are working in the cloud or in some

other form of you know, computing environment. And I think one of the interesting phase one I want to put out here, and it's well, let let's let's start go first, one step back. I think there are are two different things. One is kind of on agreed baseline, and you can set that in your company or for yourself. You can set that at whatever level you actually want. Hopefully that

level of that baseline increases over time. But I think those are you know, you can look at classical engineering KPIs how quick are you to recover from a failure in production? How quick are you able to in what's the velocity in the team, what's the error rate? You know, there's the Dora metrics out there, there's a bunch of other ways to measure that. Whether you really measure that or just like gut feeling type of measurement almost doesn't matter.

But like, that's the baseline security Laaren. You had an example at the beginning, like you know, okay, how do we you know, instead of increasing your password rotations, can we actually decrease the amount of passwords we have in

our systems? Right, It's a different type of approach. How you can think of that, And that's I think there is I don't think there is a standard in the world, but there's agreement of those argued things, right, And I think then there's this other part which is not this baseline. It's it's very drives innovation. They drive to try out something new, and trying out something new in your environment might be I've never worked in a cloud, but let's

run the first service in the cloud. Even though that's a ten plus year old saying it might be oh, this is how we do things here on running a specifically workflow, and let's try and alternative way. You're not getting into childscript front and frameworks because you try another one every other week. I'm still ten years ago. Who still?

that we're going to consider important. And you know, maybe that's going to change over time, but this is where we're going to start, and everyone's on the same page.

scenes doesn't work anyway. So that sets back to the sports andology. But I think it's it's really what is the team coming out in terms of and I think will do To your point, I think there's a few things that are true almost no matter where you are. I think you don't necessarily want to ship bugs to production, right, so less bugs and production is probably a metric that everyone agrease.

Don't work like you work in your usual environment. Just like have a hecka on type of experience, or have one person go off on basically an island for we can come back and have this zero to eighty percent solution out there. You're not going to break anything in code, you're not going to make any customers unhappy. But it allows you to move fast. So basically a little bit of an unconventional approach. So yeah, shipping bugs into production, it might actually there might be some use cases around

you know, prototypes, early things. Then you haven't figured out the exact park market fit. That's it. I still argue that you always scale down on features and have the right quality because otherwise, when you ship something of these little features that are all buggy, you're not going to make product market fit.

so you always need to counterbalance. But you know, there's this this interesting discussions of not sure if I want to go there, but like if you're doing the second or third like even when you run AWDs elastic kumunated service, but then you need the kuminat is out of lifetime, right, they need to move to the next version. You can close your eyes and hit that upgrade button in place upgrade and hope for the best and you might be lucky. But it's a little bit like I don't feel confident.

You know, there's there's order is coming in every other second, right, I mean, this is is not something what you want to play with. And once you have start having those discussions, like there's other solutions out in the world where you're not even in that problem, those other solutions might or might not be a fit for the team. Right. It's not saying we should go elsewhere, but we're definitely having active discussions around that because we're our customers are not

not buying a comment adies upgrade from us. You're buying business cards, bottles, t shirts, all those type of things in our case or in other cases might be different things. And and so it needs to come with this reality check on you know what's I mean, what's this total cost of ownership? Is the old style version of saying it probably or very mechanical way and then it come up with your business cases. But I think this is things come up constantly, and that could be this service

is constantly making problems and creating noise. You want to dive deep. Is it engineering excellence? It could also be that other partners in the business are you know, not aware of what they're causing. Like you know, think of some configuration elsewhere three layers down in the system and you have no idea, or they might be disconnected from what actually that is causing. It might call cause customer complaints and it's very very indirect and very very slow

feedback loop. And so sometimes it's not you know, the problem might not be technical, it might be a process problem. But generally this is also part of excellence, but they'll be getting a little bit further into excellence of a team, excellence of a pride team, excellence of an organization. And I think, you know, yeah, I'm definitely more qualified to talk about engineering excellence.

little bit that decided. You know, it doesn't necessarily be a dollar metric, but it needs to you know, not not sure what would other companies and teams generally have in their mind, but I would argue, you know, thirty percent is stuff that engineers need to do to maintain their status core and then you know there's this admin overhead, but then the majority of it should be to move

the product forward, to move the projects forward. That comes with coding, that comes up with his non coding activities, and really doing good for the customer. You want to increase that share of the you know, the maintenance activity.

So maintenance comes with there certain you know, in my environment's a little bit of large companies, certain things that are must do by a given date, certain upgrades, certain compliance aspects and so on, and those you know, those covering you know, ten inch percent, and they might come up with some spikes and some lows and so on, and the rest is I'm interested in the team rather spending this thirty percent time in figuring out a new service or new workload version to run, or a new

technique or documentation other thing versus. Yeah, this is what we do. Twice a year. We just do it right and think, think of something simpler than thencuminated is password rotation. Probably want to get into rhythm or rotating your password. Initially you pro might say, yeah, it's just three, let's do it manually. And at some point you have fifty and some point like, oh, this is a problem, right,

and sure give this too. Junior person to onboard to launder systems and they go into fifty places and update those. At some point someone comes in and does some automation around this, and at some point you think, why do

I even have those passwords? So I'm so there is very Yes, I'm going a little bit by feeling it by noise, but then I'm fortunately have to have access to a lot of data of thinking how many what's the average password age, what's the rotation of it, what's it over the timeline, and the timeline is very strong, like if it goes up and you actually want to go it down or the other way around, like you can still acting before it explodes. And I think that's

a really really important thing. So thinking of you know, this cuminating thing, it's not the worst thing what I have in my team by far not. It's actually serving a lot of value work. Those were things right from a real engineering perspective, right, But nevertheless, it is distractions and you're having you know, fifty plus seventy plus services and each of that needs kind of a redeployment to a new cluster if you want to move it over.

And then like even if you bring down the hour, like let's just assume one hour per service, that's fifty seventy hours per year, which is a week, giving you thirty plus engineers working on the whole thing for the year and doing that once or twice a week, it's an okay thing. But not having that in the first place, that's an even better thing. With thirty hours, you can do quite a bit of other things that might actually

drive forward. So that is, for example, we have not yet decided whether we want to get off of it or not. But what I'm excited about the team now looks at should we do something else, And very specifically, a is ECS is a very you know, natural follow up in staying on a WS. But you know, different cloud providers will have very similar real cloud and options and more managed versus less managed services.

Actually the technology we want to be using? The stack, the programming language frameworks, you know, are they actually the thing that we want to be using or should it be something else? And I think the point you made, Marcus is that none of these things have anything to do with delivering the business value on a short term timescale.

But there's like a long term implication here and an expectation that the teams are asking themselves, Hey, you know this technology you're using, should you even have to rotate passwords? You know, this thing that you're doing constantly that you may be baked into tradition doesn't necessarily make sense.

find their very own product market fit. And just look at something like S three what you were able to do ten years ago versus now, how much of an important pillar that is, for example for data lakes and other things like who would have put the data and do sequel statements on top of a three bucket? Nobody?

But today that that that's how the big data warehouses actually make money, right and and like this this is how technology was look at you know, we may chat jokes about front and frameworks, look at where they where they came from, and what are they doing today and like including things like you know those server side versions of it on how you can ventor things, and and I think it's it's we should challenge basically every day, what are we doing is this really necessarily and think

of how could we do things differently at the same time, I think there is a beauty of not classifying something just because it's old as tech dev but saying, look at you're milking the cout here right, You haven't invested much, but it still works and and I think there might there might be a servileized function out there. Yes, you might need to upgrade I don't know the library version once a year, but it still does this thing. It's not massive tech there. Would you would you design the

system exactly like that from scratch? Potentially not because the world has changed, but it's still good enough and not worth changing. Like you know, identifying those tipping points when is it actually a value or an asset and one when does it start to become a liability is an interesting thing to think of. But I would not classify any programming language or any past decisions as lessli tech there.

But you know, when you think of where's the world going in terms of you know, relational databases were that thing, and yeah, ten years ago it was still their thing. No sequels started to be other things. But these days when you hire someone off the university, they don't know how to write sequel statements anymore because that's not being taught in school they only learn no sequels and blog storage and all those things. I think it's the right

thing to teach to them. But knowing sequel it's actually good skill to have. So, you know, does my team have a bunch of relational database yes? Is it worthwhile changing that? I don't think so it is it worthwhile maintaining it to something? Very say? No, you are satisfied. Is the cost, performance, backup failures, all the whole criteria. Yes, absolutely, we need to tame the beast if you want right, but not all workloads require things on you know this

horizontal scalability and other effects. What you can actually get out of no sequel databases. So that doesn't mean just because you have this and you need to get off of it, but you need to identify at some point it might be, oh, now it's the right time to move. It might come as a project, it might come with.

effort doing this? And I think maybe this is where maybe the sort of thing that drove forward the mindset in the first place, the movement towards DevOps uh, the idea that more attention needs to be on evaluating the excellence of our of the engineering that we're doing, and maybe there's a cultural component to it that not everyone is has the sort of mindset to look at what we're doing today and actually improve it or fundamentally having to change what we're doing. Uh that that can be

scary for lots of people as well. So I feel like there's an aspect there which could have driven the DevOps mindset, which was maybe a different team is responsible for infrastructure, and maybe the sort of product teams say, hey, other teams in our company are fundamentally responsible for doing this innovation, deciding which version of Kubernetes, or whether or

not we're using Cerblus functions, et cetera. Ah Is is there an aspect where every team should be doing something in their own domain or do we think we've moved the industry has moved in a direction where a lot of these decisions are we've already identified which team is fundamentally responsible for making them.

to do it. And I think the approach today will also be different than the approaches were done you know ten years ago. Is basically sis ops and database ops and managing deployments manually and sailors and you know, local local things versus cloud cloud versions and son uh and

there is that. The way is your your ability to run it as a basically as a Pride team or as a small development team of you know, two pizza teams type of size, right the Tannish people in total ah, But no matter what model you are, you have different responses with your task is really going into these whole depthops from from you know, prop typing to delivering testing to also running it production including your you know, on call schedule, or it's a little bit more you only

you know you depending there might be some handover for for monitoring. But in both ways you have areas where I wanted to improve it could be on you should definitely be interested, like when we look at very technical

metrics error rate, response rate, latencies, certain things. You might have a little bit more leverage in one model versus the other, but you should be interesting to understanding your craft, which is in the end it is could be two ways, right, One is really just writing the code and writing that in a way that scales in terms of features, but also in terms of performance and probably many many other angles, right and you know, not testability and all sorts of things.

So that that's definitely want to but there is I think when we go a little bit you a build it, you run it model, you also have this you architect potentially differently because running workloads in the cloud, especially when you go into a little bit more of these ACYNC processing type of things. Right, do you have an event and something can happen in the back background, So think of an order being taken in into the systems. A bunch of things that are happening, but the customers don't

need to wait for that. They you don't need to wait until this thing get shipped out while the browser spends for two weeks. That's not really really right. So there is a lot of these aspects, but like, how would you architect something from scratch given the latest accourages that the cloud providers provide. That requires constant challenging because like five plus years ago it was SNS and SQS on Amazon, but now with a vent bridge, it's just like so many more features. And then how do you

do that between teams? Do you still go through either arrest or graph Well or so API or do you actually hook up some teams internally through some of those you know, cloud native event systems. There's many ways to get there, and like always can lead to excellence in the very context. But I think just saying yeah, we made this decision five plus years ago is yes, but you're missing out on you can delete these three services then just moving data around, which is solved by this

configuration over here in the cloud provider now. So I think if you're not doing these type of things and then assuming cloud deployment now and not any other type of deployments, like yes, it keeps your team busy, it keeps potentially team efficient. They might even have engineering excellence within that code, but not owning that code at all, but moving that to configuration and let Amazon who is or Google cloud or asia scale that for you and

manage that for you. That's really where scalability and excellence also it definitely comes in.

it's just one part of my job. But you know, the company also values engineering practices a lot, so there is a bunch of that filter filmed by already central things, right,

mandatory upgrades, mandatory things. Also, what we're doing as part of quarterly planning is you're showing, Okay, how much do you spend on things that someone else tells you to do on the feature side, things that you actually want to do for your customer because you understood that problem really trying to do that, and then things basically in this whole engineering excellence it's on even so I think that it's probably the wrong world we want to call that,

you know, this maintenance moving forward. So I think there is this excellent part which I think isn't is not necessarily reported out there, but I think that is something as the engineering leaders they are definitely responsible to get that into the teams and the mindset of the teams and the inn wilds right every individual there's forty hours a week. Sometimes it can be someone is passionate about the following block podcast like this one, and it comes

out with a cool idea to actually start. Others are following, you know, some news, and others are actually interested in reading the documentation on the cloud provider or SaaS provided whatever and saying hey, there's a new technique, let's try it out. I encourage really everyone to do that. That's way harder to do and get that into the mindset of everyone versus the otis is a lot of more, you know, leadership driven and saying hey, by the way, we need to do this for the long term health.

And it comes a little bit as yeah, the expectation is that you're looking to that promise space.

definition of excellence above that. So if you start it like the franchise owner, their definition of excellence is to sell as many tickets to their games as possible.

to the games. And so each one of those people have has a different definition of excellence, but they all support each other going all the way up. And then whenever you get down to that individual team player, they know that they have to catch the ball or defend the ball or whatever, and that's their metric that they

measure against. And so it's easy for them to make decisions that either support or don't support that definition of excellence because it's clearly defined with what they do and how that supports the overall organization.

become better? What's the unique training plan that they have that is or it could be also you know they bring into college and say, by the way, now we know what the competing team does, so we can actually outperform them through through the brain instead of through the body. Basically, so I think that is that to me, their verre

it starts to be really, yes, everyone does that. Everyone wants to sell more tickets, Everyone wants to have the best coach, everyone wants to have the best players and basically get the best performance out of the value invested. But I think this the excellence is not achieved by that. That's really achieving it by management, by you know, there is a lot of value and that's needed in a corporation.

That's needed in the corporational e sports and a corporation, you know, like this brand, but this excellence around like how can I how can it be innovative in my training? How can I be innovative in my nutrition? How can I be innovative in in the tactics during the game.

That's I think where versus the next level? And and also you can follow schoolbooks, you need to come up with your own approaches and more un convention or buttn't you're not necessarily your own, but at least you need to read through those unconventional approaches, try them out, and adopt them to your needs. So that that's a little bit this type of how I would distinguish that where it's engineering excellence, This is I think I was conflating those a little bit before as I went to do

the other topic. But I think there's very good like to draw a line on their excellent starts and they're you know, engineering practices or you know, other things are there to just like establish the baseline.

implications on them for that. So there's like a there's a trade off there that I can't quite wrap my head around at the moment.

of media consumption that works. And it's not just like the doom scrawling type of value, but it's actually you know, building up some some more thoughts and it's also have to space time there. But I think, you know, in terms of where you you might have been build is I think there is a space where it's just like

there's constantly noise around a team. That's when you will potentially even management will come in and they typically send That's when the consultants show up, and when management shows up and you know, weird reporting and someone looks at how many metro crasts did they do yesterday, that's a really bad sign. That's when you have underinvested. I think there I'm talking to is already like, yes, you're bringing up the point. Does is it valuable to do this

or not? Right? So we tried to sneak in those type of things at the side a little bit. I think there's certain things that are definitely of high value of you know, think of not having passwords or it's just like way simpler to manage, and it's also possible today. Five plus years ago it was very very hard to

do and these days the systems have evolve. So I think this is again like adjusting to huries the world today and you know, the world has just innovated AI in the last I don't know, it feels like cloud

three days and there's this whole whole hype cycle. Yeah, the world is moving fairly fast, right, or you know the cloud didn't exist ten years ago, that's only a decade well, or you know, depending on what's the starting point of the cloud, that's probably IBM saying the cloud exists for the last forty years and their own world

did this different definition of that. But it's really really moving fast, right and and not keeping up with that is I think it gets you into a situation where at some point a company might say, we need to stop and we need a year to overhaul our accounting infrastructure. And that's a really really hard job to do from a team perspective, from a moral perspective, from actually can you achieve that outcome? And you know, how can you keep the customers happy? And so I think this is

the balance of that. Nevertheless, I think after that, even if you don't, if you say, okay, scale down on this bucket of making things better for the sake of making it better is I think as an engineer, you just want to have that. You want to have this drive,

you want to show off. And I think this is also the opportunities that in the worlds have in corporations, no matter what role they have and what level it is, to showing off and saying I understand this technique, here's a documentation, and you know it might become the de factors standard how things are being done in the company because it comes with the context of the company. It comes with that. This is how we do deployments, this is how we run things, this is the responsibility. These

are in the incentives. And if someone is able to document it in a way or to create a create some shared code that's easy to reuse and so on, I think that's really really powerful and allows also the junior engineers to shine. They might they might actually the biggest druggle for them might be to put it out into the world. Especially when it's an internal world. It's almost harder and putting out there in the world and

nobody might see it, that's easy. But putting it out internally and you have an intelligent purpose for that, it's like you need you need some cheerleading from you know, the basically the directors of engineering or you know, your team lead to others to actually put it into a shape that is not going to annoy the others, but it's actually being appreciated. And that is really company and political context that you need to fully understand to make

sure that you can do that. But it's I think it's a big opportunity for everyone, and especially for more junior or just like very technical savvy person who don't want to deal with that, but actually would be interesting to have that better excellence across all the teams and not just for themselves.

are on my plate. It's around security, it's about infrastructure. It's about front and parts, it's about back in parts. It's about you know, potentially libraries, it's about upgrades, about deployment techniques. It's about challenging how we do run things in the cloud. It might be a SaaS provided you're using and oh be using that suboptimal and so on. It is endless, and I think as you're an engineer.

You are seeing these things and I'm here, I'm not even talking about the bugs you're seeing in your own system and the and the error reports and other things. Right, this is another part where you hopefully, hopefully that's your first home. We'll just try make sure that the customers are happy. Right, That's that's the first part where you want to use the the data in terms of oh, we had forty seven errors in this timeframe, we should probably bring it down or no, this is really kind

of the maximum. How we like we just decided in the team with with you know, senior input to know this this this is good. Right, this is really world class where you can be. But then all the other topics it is just as you and you will be welcome because a lot of those topics might not be so passionately done by many much best female as an admin task or I do the minimum versus those are really unbound and you can have an impact that is way across teams, way outside of your sphere of directly

influencing day to day. So it's both in the day to day and how do you make yourself and your team better and your softly you're directly supporting, as well as the opportunity out there of if you do things slightly more generic, share it slightly further, or you know, raise your arm on you know, becoming you know, the security expert in your team, and you'll learn all that, and then you start looking at the data yourself and to start crashing why are you doing it this way?

Start improving it into your own team, then starting sharing, Hey, this is what it is. Those are the three steps. It's that easy, Like, oh, once it's easy, others will also do it. Versus this neverless thing of you know, we might need to invest some technical spike time at some point and it's in the backlog or the back of our minds of people and it's never getting to versus simplifying it, paving the path so others can just walk it, and then hopefully if they're paved path, you're

not trying to reinvent the bill. You're just walking there because you trust your colleagues. And the combination of those also like appreciating this is one way of doing it might not be the perfect way, but it helps you accelerate. So it's both ways. It's not only sharing, it's also taking what's there.

You know, companies like that's it. You know you're as successful as you need to be for us, and junior engineers as long as you learn our systems, you're as

successful as you need to be for us. But there is still that learning aspect there, and I'm wondering if it's enough to just say, spend take thirty percent of your time to learn new things, to listen to the Adventures in DevOps podcast, or you know, use new technologies and new UI frameworks, or whether or not there's something that we need to directly encourage to make that happen effectively.

have to actively manage and track. Whether that's you know, telling them, hey, open a ticket and log your time when you're listening to the podcast or you know, or whatever the metric is. I think it has to be something that's that's active and measurable.

to one of the thoughts I haven't. And I was talking about, Hey, there's this baseline, but then comes to innovation in it, and innovation is a lot of different things, Like we talked about education, we talked about you know, you know, figuring out and taking on additional responsibilitiness on But one of those odds that have is basically, and it comes exactly where it's, oh my god, cloud is so Expensive's cost per compute is kind of a term that I came up and made up myself, but it's like,

what's the cost you're spending for the compute you're using? And you know, basically, even in the cloud, you run things on a virtual machine. By definition, you don't want this to be more than ten or twenty percent CPU, and you want to have at least I don't know, fifty percent memory free. So you're paying paying the money what you should have in your pocket, and you do your team's bonuses into the bonuses of the cloud providers. Right. This is they're charging for CPU which they are actually

not using. It's all these shared hosting environments. So that's their game, right, and that's their margin. And yet it is relatively attractive. So I think this is also very this innovation comes in. So how do you actually bring I think if you optimize cost per compute, so not purely cost, because purely cost of organization is just like turn everything off and you're done right of the business.

You will never ever have costs again. And I think the alternative if it's just like run everything on AWS, LANDA or whatever your serverlized function is, might also not be the answer. That also doesn't say you cannot use relation database. You must use down B or Mango, the B or whatever it might be. It comes way more nuanced, but cenarly, if you look at your cloud spend, cloud spend is very easy because you get your statistics over

time and everything. You can analyze and tag things and sounds, so it's quite you can have this this engineering hard and start with the data and be data driven. Just trying to iterate there. But look at the top five. They're probably consuming fifty eight percent of of your money. Right. Some of that is justified because that's actually the workload that's running there. Others such just like wait a bit, you know it's waiting. It's it's mostly the margin on

over provisioning. This is you know, relation databases you need. There's no way to scale up and scale down at least not you know, there is a way hope. By the way, we have a peak season for two months, so we scale up for the next instant and scale down again or increase the distance. So slow moving scalability can do. But those even during the days are very hard to do and if it's more spiky, almost impossible to do. But you're paying for your for your risk.

And got feeling a little bit almost like I mean, obviously hopefully you look at data, look at some requests died some low testing and everything, but like, yeah, there might be this one customer coming along and put some load on it. It's like I want to be prepared for that, right, and you need to be prepared as a business for that, and you need to be over provisioned versus the more you go to basically really pay per use. If there's nobody coming, it's zero cost. If

someone is coming, when you pay what you use. This again has at some point it has a limit, and it might be do my own thing is cheaper, right, So at some point, you know, running certain things on a huge scale is a central team that's expertise there might be the better thing for some companies or running again back to bear Metal, I mean there's just a little bit of a hype of all we run this thing on bear Metal. There are those workloads, but they

typically don't come from a startup. They don't come from you know, smallish two pizza sized teams running their own thing, even if they have a lot of requests. Those recreess are typically you know, request then do some processing, call of database, return the thing. And and like I think cost per compute is a nice proxy to say, do I have a spot where my top five or top three things? And that's just looking at the pure costs, right there comes to operational costs and other things that

you need to encounter. But at least this is hard data. What you can look at and trying to optimize for that and some point I'm good, Now I feel like the costs spent these lum and I think, you know a lot of companies might not have done that or might not have incentivized the team, and so oh my cloud costs are so expensive. But then you know, really running that very same workload, it's the very same properties. People can just spin up things and other things. You

paint that on the rack. Now, I mean community serves need to have in there to actually have for those spikes, because like scaling up and down doesn't make sense anymore. Just like put it on full load and that's what you're what you're you're having in there.

of execution for per instruction. I don't I don't know what that means, but I mean, I think there is an interesting aspect here where when we deal with the physical goods and we have a capital expenditure, you buy a thing and then you depreciate the value over time because the metal rusts or the components get out of date or you know, whatever it is about those, however it's manufactured or being consumed.

Whereas you know, if you're renting the car, you always get the newest, latest version, and if you never drive anywhere, then it's you don't pay anything, you know, conceivably. But I feel like there is a middle ground here where a lot of companies end up in and they forget that, even things like easy Too or Kubernetes.

on here. And I mean, I don't know if there's a question here, just sort of an interesting tangent.

have this fifty million dollar built every three four years

is a different type of discussion. And and I think this is as you're optimizing the cost, you're not bringing the cost down by ten dollars, one hundred dollars, one thousand dollars, ten thousand dollars, whatever your workload might be you're bringing that down multiplied by I just multiply the to a year, right and look at the yearly cost, right, because at some point, you know, things change so much that it's very hard to for long over the long term.

I think it's probably basic finance here, but also it's important that numbers team members understand, like, no, you're not bringing down this workload. You know, fixing a bug for one user. No, you're fixing a bug for one user day which is tween a sixty five users a year, or you buy I don't know whatever it is, and it's like, no, this is This is a little bit where capital investment in terms of software engineering comes in.

And you know, some of that is on a product side, but I would argue some of that is on engineering side of saying if we fix these three things, that's the impact you have to the customers, to the business, to the cost to whatever. And that's you know, if you invest an hour, now like you can basically count how many days or how many weeks it takes until the return of investment is There are very simple things

and there's very more complicated models. But like get the back of the Napkain calculation into everyone's head, like just like, yeah,

make those decisions based on that. And if you're really good at this type of investing your time where there is a return, and preferably return is on better customer service and better customer value because that really feels your business, right, So like old engineering excellence aside, but where you can invest and you know, prove improved things from an engineering side. And it has also either direct so sometimes it's indirection.

How do you measure downtime? Right? If the team was down once and once a month is pretty bad, right, Uh, all of a sudden it's not down anymore. It's probably a good thing that happened. I don't know X months in the past, but it probably didn't happen or the work change incidentally, right, it probably happened because someone's yeah, right the thing.

using a public cloud. But you know, if someone knows, I think they should ping us on discord and actually, you know, give us a topic of conversation here. I think that'd be super interesting to have someone with an expertise in on premise data center management.

like completely going off and inventing their own stuff. And I think that would be a really cool conversation to have and say, just you know, how's that working out for you?

they have some aazing processing kind of these servilized pipeline type of needs. I feel like if they were running and I'm guessing here is krumen it is really in the cloud in terms of how potential is a cloud provide to sell it and the potentially the options that were there. I think that happened a three years ago when they announced it. It's different than oh, like you go all in on the cloud and figure out how can you you know, costper compute, how like cost per

compute we're not good for them? How can you squeeze it out? I think there might have been a cloud way to do it. And those are you know, top notche engineers. I'm pretty sure they looked at a bunch of things. But this is also one of the arguments like why you know, I was you know, looking at I don't want the krumanides upgrade a year, but if it's there, it's not the worst thing, right, But there

are other things like kumunides. You know a lot of people put the stamp on a LinkedIn profile because they want to have the stamp. It's the wrong incentive, right, and it's but it's also a huge complexity to learn. I mean, learning cumanities is equally challenging us learning ITWS and learning Google Cloud or whatever it is. Right, It's a huge thing and really really mastering that and then scaling it to your needs and getting the right cost

performance security profile. That's another undertaking versus most teams might actually just like, yeah, we have the kubanities in the cloud and fully managed and these are the seven settings we're looking at. And if you're doing that, it's probably like something like you know, a more cloud native saying that it is more or higher higher managed service like NAS has easy as it's just a cluster, has a name.

That's all you can configure, and then you'll say, oh, here's this service, here's the doctor instance, and that's giving me so much CPU and memory and then hopefully that matches the wallet you have and the workload you're thrown against. It has the lot of right balance, but it managed this deal all the scaling for you, Like okay, you need to configure that right, but it's like it does

all the magic and the nine thing about it. In five years, if you never touch the code, okay, your own code might run into security volunabarities, but the underlying hardware has changed seven times, runs faster by then, has been gone through innovation cycles by the cloud provider, and while doing nothing, so your code gets more secure and faster while doing nothing. This is a really really beautiful thing to have then using managed services, and I think

that's something that's undervalued. And often like you know this on ITA, that's not sure how much we can go deep there versus out like there is you back it by your own virtual machine and you're only easy to instance versus by fargate, which is basically easy to fleet managed by WS so you don't see them is looking at a one to one comparison. Obviously back by easy two is cheaper because you manage it, you have the responsibility.

But they also know there's so much free capacity there what they can basically what they are charging you because you have to manage the capacity versus you're really only paying what you're needing. So again it's yet one level higher. So depending on where you are on the managed services, go one level higher and you get a bunch of things for free on maintenance, security, upgrades, maintainability and like again strive there, like what's your next player? Then after that?

What's your next player? After that? At some point I think there are two stopping points. Some point Okay, I don't understand this thing anymore because it's too abstract. There's services out there, drop this thing, and we kind of run it right, and you want to have a little bit more control depending on your needs. Great for startups, great for prototypes, great for the amoor wasn't great for certain workloads, but not lessening everything. So that's one stopping point.

And I think the other stopping point is when you don't it's not a managed service, but the manage service manages you kind of think it is. At one point, at some workflow characteristics, they might become expensive. A very classical one is the NAT Gateway. Just put it up.

It's very cheap, but once you actually hits traffic at some point you think, should have run my own, right, And those are things that I you know, I'm there going back to the Hay conversation, is my yeah, should I run not their mental arcy, but should I run my own here? And it's a slippery slope, right, I mean it's from a cost performance cost professor, absolutely, it's

probably also not that hard to run it. But then comes the whole security maintenance, and nobody will have a clue of how this works because there you need to go deeper into networking and to routing tables and all the other things. What you need to understand. I don't necessarily want to spend that time there, but I fully understand companies that don't run this nap game because it's just like hitting a limit where they want to spend

the money. Yeah. So it's just one example, right, So, so I think even managed services they are not this is the way to go. There is orders, what you need to set and hate by they here's here's where we stop or here. Excellence actually means not doing the thing that sound.

It's been absolutely fantastic. Everything I want in a tablet for going on vacation with.

I mean, every once in a while I need a game to actually entertain me on a fourteen plus hour flight, which I don't recommend to anyone unless there's something really good in it for you. So that that's really my my core requirement there, which is why I've gone for this instead of something with like E yanch or some or on that approach.

every second week. I think every second week, and I'm following along almost everything, and I think it's it's just inspirational. What you know, talk about engineering excellence. It is what are other disciplines doing right? Sometimes it's someone in psychology some other science feels investing sports everything and like just gives you a different perspective learning from the best. What they have already figured out is what his tagline is. And I think I really really like that. And I

can only recommend if you are short of time. I don't know how much I should go into recommend recommending other people here, but it's clear. Thinking is the book Shane Heart brought up half a year ago. It's summarizing about the whole community what he has built up over the years and the consistency definitely worth a read right on.

or not. It's just there's entertainment there for people of all types. So that is YouTuber The Vocalist and her reaction of Metallica song one, and that's all we have for the podcast.