Us going on. Everyone, Welcome to another episode of Adventures in dev Ops. I'm your host for today, Will Button joining me in the studio my co host back for what is this episode number three? Warren, let's call it three. We'll call it three. Yeah, right on. I'm excited. It's starting to look like a trend here. Yeah. You know,
I'm Warren at CTO at Authors. Thanks for inviting me on. I know I said I promised just one or two episodes, so you know, I think this trend is going in a good direction, and I hopefully will be on future ones as well. Cool. I hope so, because I'm enjoying having you here on the show and joining us. Also in the studio today we have the DevOps app activist from Dina Trace, the Cloud Native Compute Foundation
ambassador and host of the Pure Performance podcast. And I can validate this because if you're not familiar with the podcasting industry, there is actually a secret handshake used by all podcasts hosts around the world. And I did validate this, so I can confirm that this is actually the legit non chat GPT. Andy Grabner joining us on the show. Welcome, Andy, Thank you so much for having me I know that AI technology is pretty good already, but I'm
not sure if they if they can simulate my accent. Yeah right, And if you're joining us on the livestream, you can actually see Andy as well as Warren and myself, and we can do the test as well to show that it's a real legitimate background not behind not just an artificial background behind us. My my leads move here. Yeah, I can walk behind my kitchen counter, but I need to take off my headset quickly and I'll show you right, Yeah, same thing. There you go. Perfect successful test can
confirm that we are not a robot. Cool. So welcome to the show. So you've been doing the Pure Performance podcast for seven years? Is that right? Yeah? I think it's about seven years, even maybe almost close to eight years. We had last week I believe it was the two hundred and first episode. Oh nice you have released. Yeah, and yeah, it's been. It's been exciting and it's been if you look back, it's
actually quite astonishing that we've been doing this for that long. And with we, I also mean Brian Wilson, my Cobles them myself, so Brian, it's not here today, but yeah, thanks Brian, Right, so for anyone who's not familiar with your podcast, give us the short summary of what you talk about on there. Yeah, So we initially started it because both Brian and I we've been in the performance engineering space for twenty plus years now.
It's been a long long time we've both worked at Dana Trace. Even though the podcast is not related to the product or the services we sell, but it's basically based on our experience and what we see out there. So we started initially to talk about what are the performance problems that we see in modern day applications. You know, why are applications crashing, why are applications
slow? What are the most common things that we see. How can we educate our listeners which we initially thought of mainly developers and our and performance testers. What can we teach them? But over the seven years things have changed, and because our fields of topics has changed and evolved, and we moved from performance engineering to develops, to cloud native to kubernatives, to platform engineering.
We covered a lot of range of topics and we also had over the years many interesting guests, a lot of practitioners also people that you see on big stage in the IT So, yeah, it's been quite interesting and I think a broad range of topics and hopefully there's something in there for everyone. Right on, So you mentioned that you said your podcast audience initially was performance
specialists and practitioners, and you've seen that change over the years. Yeah, because also we changed, right and with we I me and Brian and I we have changed. And the reason why we changed is because obviously we you know, day in and day out, we deal and we work with our community that's the dial traized community, the opstability community. And because our product
and capabilities have evolved, so have the people we interact with. They've changed, and we see nowadays more and more you know, enterprise architects, We see cloud architects, We see people that are responsible for big Kubernators clusters. We see people that are very heavily engaged in the CNCF community like I am in the Cloud Native Computing Foundation community. So we're inviting a lot of kind of my counterparts to the podcast and discuss topics that are relevant in the Cloud
Native ecosystem. And so yeah, that's why it has changed. But we always try to come back and kind of relate it to performance because in the end, we all want to make sure that we help the global community to build performance and resilient systems. What was new eight years ago? Like, what was the topic of conversation that what was performance important? You know, what were you talking about back then? I think episode number one, two
and three, or at least the first handful. I believe we're all focusing on what are the top performance problems in Java and dot net applications because these were kind of like the two predominant technologies we were still dealing with, right and we still are seeing a lot of Java dot Net, And because Brian and I were doing a lot of load and performance testing and a lot of performance analysis on these type of applications, we said, let's discuss what are
the key top the key things we find in applications in Java and dot net applications that make applications fail crash, be slow and talk about it and educate our listeners. So you have a secret love for the CLR, then yeah, I would. Maybe the word love doesn't come to mind as a birthday sure, yeah, But I mean the interesting thing is, right, it's in the end, everything we discussed and all the patents we found, Let's
say most of them are not limited to a round time. Well, let's say you only see certain patterns in Java or dot net, because patterns are patterns, and they happen whether you are developing. We just did an episode recording that I think will come out next week on mainframe, right, so we also cover mainframe, and we see similar patterns that lead to problematic software in the mainframe versus Java, dot Net, and now going all the way
forward, things we see in the highly distributed world of kubernets. So same patterns. I want to ask about that real quick because one of the patterns I've seen throughout my career is a tendency to blame the stack for performance issues. Like a really common one is, oh, we wrote this in Java, but that's not performance, so we're moving to dot net or you know, insert any other any other technology you want in there. You have you
seen that? Yeah, I mean the easiest for humans is to blame somebody else, right, And the reason why that is, I believe is because we as an industry, I think we try to make it as easy as possible for engineers to write code. And easy means we are abstracting things away. We're abstracting the complexity a way. And therefore, when you abstract things away, you allow people to build things very fast in a generic way,
but not optimized or optimizable for the specific use case. And so when people say this would be fifty times faster, or if I moved to random x or y, then I would argue that this would only be This is only true if you actually change the implementation underneath, right, if you are using you know, if you if you're replacing the Hibernate with the A Development Entity framework, you will still run into the same problem because these are all very
generic frameworks. But if you don't use them and optimize them for your use case, you will still end up with the n plus onequibit problem capturing and fetching too much data that can gets into memory that again gets filtered in Parston memory, which leads to too much memory usage, high garbage collection, high CPU bad performance. So my answer to this is, if you want to build really high performance applications, you need to understand what's actually happening in your
stack. Right, How you know what type of data do you need, where you get the data from, how do you get inefficientcy what type of data do you even need because the otherwise you will end up with the same problem in any run time. It doesn't matter. Yeah, the best case, even if it does fix whatever hypothetical problem you have at that moment, you're probably just creating a whole bunch of other ones that are not seen because of the swage, right, Yeah, yeah, I agree with you.
And you know, I think we are living in an interesting situation with now. When I kind of grew up, sounds like strange. When I went and I got educated on software engineering. I started in high school. I was lucky enough to have an education system here where one of our local high
school was specialized in software engineering. And we learned assembler, we learned see, we learned the core, and then we we learned like how to move bits around and and you know, and how to how to manage memory yourself and and these are traits now that that I think are probably I'm not sure if they're still taught, but I think the necessity of if somebody comes in new and wants to become a software engineer, especially because of such a high
demand, I will probably not start with the assembler. I probably won't start with with managing my memory myself. But I will start with a higher abstraction because I want to be productive. I want to contrue you, and with that it means that you know, I may miss things and I do things that later on will lead into a problem because I don't know the underlying stake
and what's really happening. I've seen the problem go the other way though, right, like starting to play directly with threads or whatever the underlying is because you think you can do it better. And maybe today it is true, especially in some larger organizations, But then the industry or open source or whatever the language you're using framework of choice evolves over time and comes up with a much better agreed upon strategy than whatever you're using. You must have seen that
as well. Yeah, I agree with you, And I mean trading is another perfect example, right, I mean that's in many horror horror stories or problematic source. And by the way, I don't want to make the impressional gift impression that I know how to really in the highly efficient code. I've just seen many examples from other people that have built it, and I could help them identify those issues. So if you listen to this, don't hire
me to build that system. When you say when you just you know, everyone's on the same page, when you say you know performance or high performance, are there sort of numbers that go along with that that give us a good viewpoint into what that actually looks like. You know, this can be anything from systems that can handle i don't know, thousands or millions requests per
per minute, per hour, highly highly transaction volume. But this can also mean systems that have lower transaction volume, but that need to process terabytes of data. You know, patch processes if you think about it, and patch processes have to finish in a certain time because otherwise you may run out of a window of opportunity. Or there's even regulations where like especially in the finance and the energy sector. Right now, I've just worked with the cloud client
there. They have to finish processing all of their transactions, do it in the patch at a certain time because otherwise they're not allowed to trade the next day. And so this is this also this this may mean they don't have let's say millions of transactions, but they have transactions that are bigger in data that needs to be processed and validated, and you know, external systems need to be contacted and updated so highly performance systems really means whether you can deliver
what your end users are expecting from your system. And end users can be real users that try to book an online ticket at a show that is highly on demand, or it can be you know, other systems that call an API and expect when they call you API that it will deliver the result in a certain amount of time, and that time could be seconds, but it could also be you know, maybe an hour. But this is certain expectation
to a system. You triggered my curiosity there with the regulation, like there must be different observability importance that happen in as something like that, where it's not about necessarily the reliability of individual pieces or requests that come in, but over the whole data set. Like, that's not a world that's really familiar with me. So any insight you got there it would be super interesting.
Yeah. I mean, the one thing that comes to mind here, it's really we talk about data integrity obviously, right, I mean from an obstability perspective, if it's not just that the indust as you said, individual pieces of code that get executed to do certain things work, you know, I mean, obviously they should work reliably but in the end, the ends to
end business process needs to produce a result. So, for instance, a very simple example, if I would wire you money, then I want to make sure that if I wire you money and it goes through, it ends upon your bank account. If something great, If for whatever reason you know, something happens in the middle and something crashes, that the money doesn't just end up some where in a random account. So I think this is also
very important that we think about end data pipeline observability. This is another big topic business process monitoring. There's different terms I think for these things, but in the end, you want to make sure that whatever you do, you end up with the results that you expect, and that proper resiliency and proper error handling is taken care of, and at the end the data on both
sides of the system is valid and accurate. Makes sense. So you mentioned that some of your listeners are architects, and it sounds like some of the performance conversation has been shifted from reactionary to to further up in the design process. How do you define or what are your thumb rules for design finding a performance application without getting into premature optimization. It's a good one, and then Wren is smiling. So I think what I learned, and I remember what
I said earlier. When I went to high school, I was educated as a software engineer, and we were never educated unfortunately back then, to think about how to measure and observe if my software is actually doing what it's supposed to do, because it was in the mid nineties and sure be road log files, but we didn't never think about metrics and traces nowadays and things like
that. So what I can advertise now or advocate for, and what I see is observability as a requirement, obserability driven development, which means if I'm building a new service, if I'm building a new app, if I'm building a new function, whatever it is, I should not only care about what this function does, but also how I can validate and measure where the function,
you know, execute in the expected time with the expected resiliency. So that means what happens if something, if if my dependencies fail, how do I react. So it's really about thinking with observability as a non functional requirement and because first of all, it helps me as an as an engineer too later on validate that my stuff actually works. Is expect that it helps me if something doesn't work as expected, that I can troubleshoot if I have the
logs that traces the metrics that the need. But it also helps me to justify my work because if somebody comes to me and say, Andy, we need a new social media platform. Right, we have already too many of them. But let's say social media platform, and what's the measure of success. How do we, you know, differentiate ourselves with the others that are out there. So we want to have some metrics, and one could be better user experience, faster posting, easier API, so that others can integrate
better with my social media platform. And then I need to say, if I'm developing it, I also want to make sure we can measure that the system is really used. And the hypothesis that we have that our system will be adopted better because we are faster and more resilient, it's actually, you
know, the hypothesis works. And so what I see now more and more is this, you know, pushing observability into the minds of architects, of developers early on, of people that build frameworks, of people that build new libraries. That observability is baked in and not an afterthought. I think that's
very important. Yeah, for sure. I've spent many decades at this point working in early stage startups, and I think that's been one of the most critical mistakes in a lot of the startups I've been involved with, is failing to define what I call it the success criteria. So you build this product and you launch it, but then you don't have this definition of success to tell you that your product is being being consumed, utilized, and implemented the
way that you thought it was going to. And so now you're off building your company in this direction, and your potential customers are off going in this direction and you never identify that gap, which ultimately is bad for your startup. Yeah, exactly, because and especially at the startup, you have obviously limited resources, and you need to make sure that you are putting your resources
where you have the most impact. But if you cannot measure the impact, and if you don't know where what's your north star, like where you need to go, then yeah, and you're just running potentially in circles. Yeah. So you run out of cash, yeah exactly, and then if you're lucky enough, you sell whatever you have left and start something new and a
serial entrepreneur. It's also great. I mean you run into the problem sometimes there though that you're successful in spite of your mistakes, and then you get to tell a great story about all the things you did right and then people go and copy you. True. Yeah, yeah, yeah cool. So in your work over at the the C and CF, what does a what
does a typical day as an ambassador look like there? And how does that relate to your podcast, because I think there's probably a correlation there of your public presence versus or your public presence aligned with an open project like C and CF. Mm hmm. Yeah. So the first of all, I'm really
grateful that the CNCF gave me the opportunity to be an ambassador. And just for those people that don't know how this works, you have to apply to become an embassador, and investador really means to work the community, to help the community to grow, to help new members that come in, to find the way around, to help them contribute, become just part of the community, right. And you also have to reapply for the investadorship because it's always
a year. So we just had to reapply and now we need to keep fingers crossed that we did everything right. Now, how does this relate to my podcast? Since I've been an ambassador, I've obviously been more connected or better connected with project leaders in the c and CF. And for instance, we just had an episode of pe performance around the open source project kepla a C and CF project. We also just recorded an episode on our Goal,
which is a very prominent player in the C and CF. And so me being an embassador gives me the chance to easier connect with these different project teams, with the project leads, with the maintainers, and then I offer them platform to promote their work. Then also hopefully help with this grow the CNCF ecosystem and the CNCF community and educate them. It's a win win situation because in the end, you know, if you look at the CNCF, it's
a big, big landscape. If you look at the CNCF landscape, our industry will try to figure out how we build platforms on top of Kubernator is the base platform, and then picking the right tools and so my goal is to educate on how to build platforms in the end that make people successful. So platforms that allow developers to push out code faster. The big topic for us is always observability. How can we make sure that everything we build on
top of kubernatores is observable by default? What does this them mean for rgo, for flax, for open feature, for keplar, some of these tools you know that I had represented on my podcast, and so we educate people on how to choose these tools, but also get observability by default when they when they decided to choose these tools in there in the landscape, right, Because to apply to be a C and C of project, there's there's criteria
you have to meet, right, and there's like an incubation process, and you have these certain goals that you have to meet in order to get to that graduated status. Is that correctly exactly? You start with the sandbox, so that's kind of like the first phase of it, and then you need to prove that you have real adopters, meaning that people that are really using your project in a production environment. You need to have people that are maintaining
the project, that are contributing to the project. So you need to provide you need to kind of prove that this is a viable project that doesn't only depend on a single person, because that could obviously be if somebody is very enthusiastic and build a great framework or a library of obcheck. But it's it's one person. What happens if this person is hit by the legendary bus,
right, So that's that's uh, that's why you want to. You want to build an ecosystem and a community, and you want to you need to prove it. And once you prove this, then you get into the uh uh, the incubation status, and then into the graduated status exactly. So that's how the scene CF defines the different maturity levels. The details of the criteria I don't know them. I cannot recite them by heart. I'm pretty sure there's a website we can link to get a tattooed on your forearm.
Oh yeah, I think there's actually a sci fi movie waiting to happen there, you know, because we talk about the bus factor. I bet we could make like a Stephen King style sci fi book or movie about this haunted bus that just travels around the world taken out solo founders. Huh interesting, Yeah, side quest, it was just a tangent. There's the XKCD with you know, a small little self contained component that everyone's depending on that's holding
up a mess of architecture. So I think I think you would steel like society, civilization Crumball very quickly in that regard post apocalyptic film for sure. So how did you get How many years have you been a cn CF ambassador? Uh, well, I am now applying for my second year, so it's still in my first year right on, and uh I'm still a baby, I guess in that, yeah, in that respect, but yeah,
no, it's been, it's been. It's been good. And as I say that, just reapplied and so we'll see if they will leave me in the game. They also so with data scenes you have, they invested the program. They also have we have regular check ins where we basically show what we've done in order to contribute back to the scenes safe in order to qualify to be a good investor and not just use the title for nothing, you know, besides doing the podcast that do speak at different meetups and conferences,
especially KCDS, gubernators communities these days. The topic that I'm currently kind of promoting and advocating for is the whole topic of platform engineering. So I was really lucky so then I think also the Investador title helped a little bit to be picked as a speaker at some of these local events that we have here
in Europe. And yeah, so yeah, so hopefully hopefully you will have me back at some point and I can then you can ask me the question and say how long have you been an embassador and I can say seventeen years now, and then I will ask you why did it take seventeen years for me for you to call me back on that podcast? Well? Currently I can use the excuse that my emails down, but if it's down for seventeen
years, I think we'll go fully off. Is it really a problem though, if my email doesn't get fixed, as long as slack and everything else works. How did you get from I don't know if your podcast is related to this, but you know you started the podcast almost eight years ago and now you're CCP ambassador, Like, what was the path here? The path
here? It's a good question. So when we started the podcast, funny story is that we had a very close friend, one of mine, Brian's close friends, Mark Mark Tomlinson. He actually encouraged us back then to start the podcast. He had his own podcast was called pup of proof Fites. He also talked about performance and then he said, hey, why don't you just start a podcast? And then Brian asked me, He said, hey, do you want to do it? And what does it take and how
do we who do we need? Who do we need to ask for permission? And I said, you know what, let's just do it and ask for forgiveness later in case somebody is not happy with this, and that actually worked out pretty well. So we we just started with pure performance and what we what Brian Again, thank you Brian for all that he's doing, because I just need to talk and he's doing all the the post production and everything, so I mean it's not just talking. Also, try I bring the
guests because of my network with our community that DNA Trace the Scenes. You have community, so I have good connections to always find guests. But we what we tried to do is and what we kept is since we've been doing it, we do a podcast episode every second week. This is also what you need for consistency. I'm sure you guys notice as well. You need
to have consistency. This will help you in recognition and this will help you I guess also with all the different algorithms that then eventually bubble up your pots guests amongst the millions of artists that are out there, and uh, yeah,
and we've been, we've been. I've always found the most the most interesting thing for me was that I always try to pick guests where I definitely can learn something, right, because I want to pick guests that are experts in the field where I don't have expertise in because this allows me to learn something. So at least the hour that I spent in recording the podcast, it's an hour well spent because I learned something. Whether we have listeners or
know. I know we have listeners, but that's that's always good. And yeah, we just pre COVID, I was traveling a lot with different different conferences, meeting different people, and so when I thought, hey, this is an interesting speaker at this conference, talk about an interesting topic that is adjacent to the performance kind of main thing that we have, then I asked
them, do you want to become a guest on my podcast? And that always help and so yes, eight years later, we had, as I just said earlier, the two hundred and first episode, and we were lucky in the early days to have people like Guaranca. She was the head of performance inside with ability engineer at Facebook. We had Kelsey high Tower, We had Jean Kim talking about develops right. We had many amazing guests that I would have probably not had on the podcast if we wouldn't be persistent with,
you know, kind of building up a reputation. Some luck. Obviously, there's always some luck that you happen to be at a conference and you happen to meet this person, and then all of them are really helpful and they are willing to get on the podcast, and so yeah, so this is how It's how the path kind of went, and here we are and now I'm with Will and Warren. I mean, what else can I ask for? I don't know. I kind of think that's the top that I know,
Like, where do you go from here? And I think, I think I just follow your lead and I shut down my email and retire, right, go off the grid by a typewriter and a cabin in Montana. Yeah, he sud He was super motivated to talk about platform engineering. And I feel like I've heard this term throughout my entire career, and to this day, I still couldn't tell you what that is, so, you know, funny enough, I'll give you my my explanation a second, but funny
enough. I was on a was it It was another podcast. It was similar to this, but it was like a discussion where it was we were two and two on each side and one we're saying DevOps is dead, and the other ones were saying, no, depthops is not dead. So kind of like defops is dead, that platform engineering is the new thing, and then the others. This is where I was, depops is not dead because I have you know, been talking about develops for many many years, and
so they put me on that area, on that side. But coming back to platform engineering, I think platform engineering is trying to solve one big problem, which is that currently when you try to build software, tested, deployed, operated, you need to know a lot of a lot of things,
right, a lot of different technology. The technology we work on is very complex, and especially if you then try to scale this if you have if you're not like a small startup like the three of us, but if we are one hundred five hundred, one thousand engineers in an organization and you ask from every engineering team to do everything end to end, then you probably end up with a lot of duplicated work, with a lot of wasted time because
everybody tries to figure out how to build, how to deploy, how to patch, how to do security, how to do network and so platform engineering, from my perspective, tries to provide what's called golden paths. So as in an organization, right if you are I don't know if your job.
If you are an organization that is heavy, Let's say I'm building Java based applications because that's your skill set, then you provide golden paths where developers can go to a portal and say I need to build a new travel based micro
service or a new travel based app. And then the platform engineering team provides a self service portal whereas a developer can go in, I don't need to care about how it gets built, how it gets deployed, and don't need to care about security because all of this is bake them by the platform engineering
team. So the platform engineering team is essentially building an internal product that solves problems of internal customers, which are developers, so that they can focus on writing code instead of having to figure out how to package the code, how do they push it out into the different environments, how to do automated performance and low testing, how to do security checks. So platform engineering, in a simple sense, is fulfilling the promise of DETHOBS and enterprise scale by providing
it as a sealf service. I think it's a longer nnger than I anticipated, but I would like to add that a little bit too, because the
way that I treat platform engineering is it's about putting guardrails in place. You know, because every company has financial, operational, security, and legal constraints about how they operate their business, and so I treat platform engineering as a way of putting guardrails in place so that someone from the engineering team can get from their application idea to production while meeting all of those constraints, even when
they don't know what those constraints are. It's sort of like, you know, when you go to the bowling alley and you put the bumpers and the gutters along the side so that no matter what you do, your ball is going to make it down and at least hit the pins. I kind of treat platform engineering as the same way we're putting in the bumpers to keep your bowling ball from falling off into the gutter. I like that. Yeah, yeah, it's a great analogy. I may borrow it at some point.
I'm pretty sure I borrowed it from someone else. I just don't murder you. And I think maybe one other thing the platform engineering is a platform engineering doesn't solve one hundred percent of the problems that you have in your organization.
I think as platform engineering teams, you should, like with any product that you build, right, you want to figure out what are they the pain points of eighty percent of your engineers, whether eighty percent of your engineers waste their time and things that they shouldn't waste the time on, and then come
up with an easier way for them to do it. And this could be something simple, as you know, create a wiki page to explain how to create a new project with at least the basic security scans, but it can be something as sophisticated as building an IDP and internal development platform using tools like backstage, port Io or others where developer just clicks on the button and then fills out a couple of fields and then outcomes a fully configured Git repository with
pipelines, with security scans, with observability building and so I think the point is, you know you need to, like when building any type of product, you first need to understand who is your end user and why would you end user use your product? What pain do they have that you can solve? And if you figured that out in an organization, then you're also set up for success because the other way around will be somebody here as platform engineering
is the cool thing, so let's build a platform. And then you build a platform, and then the platform doesn't help anybody in the organization because it doesn't do doesn't solve problems that their people have right now. No, I
think that's a really important point that often gets messed. Like if we're talking about platform teams as a team topology concept, then what's often missing I can't believe still today is what you're essentially saying is product management from that platform team so that they know what they're working on, working on the right things.
And this is also why what I try to stress coming back to savability, because that's what I do on a day to day basis with any product, whether it's your let's say, your your e commerce website that you're building, you want to monitor how many people are using it and how much money.
Do you make the same issrue for your platform? If you're building a platform, you want to know do you have one development team that uses it because you feel sad for you if they don't use it because they're your friends, what do you have eighty percent of the development teams use it because you actually scover a problem and are they becoming more efficient in the work. So that's why you also need to observe success the adoption platform, and therefore you need
to solve a problem. And very importantly, your platform, it'sself ulto needs to be available at the time when your developers need it. It needs to be resilient, and it needs to be performing. Because if at eight o'clock on Monday morning and all of your developers come back on the weekends and they all of the great idea, they all start writing new stuff and they use your platform and say avenue adear, and all of a sudden, your platform
crashes, and you also miss the point. So you need treat it as a product, and therefore you need to buy all the principles of software engineering through that product. Yeah, for sure. It's one of the big misconceptions there that most of you have addressed is that your engineering teams are your customer, and the misconception is that they don't have to choose your platform as a
customer. They are free to go elsewhere. And so unless you have that frequent communication with them and really understand what they'res are, they will go somewhere else exactly. And you know, I've seen the problem actually even go further than that, whereas like if you just count the number of teams that are using your thing, you don't know if they're actually getting the value out right. If your platform is going to fall over one day, you know,
that could certainly be a problem. I've seen organizations mandate the use of internal tools that have been built which can slow the teams down and whatnot. I mean, I assume the list of problems here goes on and on. I'm sure Andy has quite a few number of horror stories. Yeah, And I mean that's I think this is what I said earlier. It's the if, I mean, certain things in certain industry have to be mandated because there's no
way around. You cannot just allow I don't know, somebody in the financial and highly regulated areas so in governments to just pick any cloud vendor in any GEO, that's just not possible. So certain rules of engagement, certain cardhels have to be there. But if you come back to understanding the pain point of your internal customers and then making their life easier, it will be success. If you think you know what your customers, your internal customers need without
even talking to them and building something, chances of success slower. So it's as simple ast that. Yeah, and that's a hard conversation to have, right, because you're going to go up to someone and it's the same for all areas of your life. You know, you have to walk up to someone and say, hey, what am I doing that's actually not helping you? Or what can I do to help you better? And so you're going
to get that feedback that's just going to like ship away. You know, you walk into the conversation thinking, damn, I nailed this, and then you have this conversation and they start chipping it away at it and it's feedback that some people don't want to hear, but it's absolutely critical if you're going
to have a successful platform initiative when people really like negative feedback. Right, So I just had the pleasure We just had our annual user conference from from Dina Trace two weeks ago and I had the pleasure to have Marcellina from Dell on stage and he was also you know, he was talking about how they brought observability as a self service into their internal platform and I asked him, what are the measures the KPIs that you were reporting back to your leadership that
actually shows that your team, your platform team that is bringing up stability in is actually successful manage any impact. And and I think I need to look at the actual numbers, but it's all the recording is available. But one of the things that they measured is they were serving their engineers and say how much time do you currently you know, spending coding so doing things that is considered toil. And they went from something like thirty six percent productive to eighty
percent productive. Wow. Right, because they received a platform or they were given a platform that really solves problems. That was you know, taking away work from them that could be automated by making it easier to get their logs, metrics and trastic without having without them having to figure out how to whether write the log how to create metrics. They just Marchia on his team. They just invested a lot of time they were investing in in invester pipelines,
in more resilient pipelines. So he showed some numbers on how many million of pipeline runs they have, you know, increased in a matter of a year or two, which means faster turnaround, faster faster software engine, faster software development and experimentation. So it's really fantastic. And so this is like the reason why I asked them, what metrics did you use to show us success? I was really blown away by you know, the toil how much that
went down, the number of pipelines they ran. That means how many more releases they pushed out. Also, the I think he had one interesting metric game we are coming from an opstability perspective or background, the meantime to instrumentation, how long does it take you to actually get the instrumentation? And uh, they it was fantastic, right, good twist on the original Dora metric. Yeah, yeah, yeah, we get the metrics right after our first
production outage. Do you measure do you measure by the number of negative tweets or what do you do? I mean, you know, I've heard a number of companies though they talk about gray failures that are ones that it's not so obvious that there's a problem and that requires external feedback that a set of serial requests or something like that achieve too high a latency, for instance, and so having that stream seems super valuable to be able to know immediately by
watching the API from some third party social platform as an observability tool. Yeah. No, I mean yeah, I mean I was half choking earlier. I know that what I was saying is I hope that you have more observability
than just monitoring the social feedback on the feedback on the social media. But yeah, I mean we I know, we have users, customers of managers and they are pulling in data from the Twitter API and winteresting there were other API social media to get a kind of like a sentiment of what's going on out there. Wow, that's going to be Elon's new monetized product is using x as an observe observability platform. That would require people to still be using
it. It's true. And I stopped the commenting here because run over. We all have careers on the line here, So next topic please, Yeah, and awkward silence. No, uh, you know, it's really interesting. I was actually listening to some of your more recent podcasts to see how the how it's changed over time, and one of them really intrigued me was the one on AI coming up, and I figured it'd be a topic, and uh, you know, whatever interest you have or insight you got,
there is certainly appointment for whatever is happening in Double today. Yeah, I mean AI is a topic. I think we all need to figure out what that means going forward, right, because we were all I think, at least from my side, I'm both excited yet it's not scared, but I think we need to be very careful we probably do with it, and I think that's that's something we as an industry, even as a uh you know, as humans, we need to figure we need to figure this out as
a society. Now from what I believe where AI is obviously great help as you can tell from my accent and for my not private English. That helps me a lot when I write things so that I don't make grammar mistakes, because you know, that's what this is great for. It also helps a lot of people to write code, not coming up maybe with new cool algorithms, but at least writing boilerpled code. And otherwise it's just a time consumer.
Right, And how the question is right if we are letting if we are ending up creating and letting AI generate a lot of code, and then nobody understands that code anymore, and how can we tell if that code is not doing something that we didn't expect or even in the future something malicious.
Right, Because if we had a one of our keynote speakers at our conference, she was heavily involved in the I and EI research, and then the security aspect that she was saying, if hackers infiltrate the training material of AIS that generate code, then who knows, maybe THEI generates some backdoor into your code that can then be taken over by somebody. So this brings me back
actually where I started my conversation in the beginning. If we are trusting the stuff that is given to us without understanding what's happening, then we may end up with the next big performance problem, resiliency problem, or security problem. I mean, for sure, we know that it's possible to poison the mL models so that the result is problematic. It's obviously short jump there to get to the next step where you could do it in a way which causes a
real problem or opens the back door. Like you said, it's a really good point. Do you see it happening in like the observability space, Like is there just anominally detection or are there things on top of that that you see coming down the pipeline that really are a real step forward that you're looking
forward to. Yeah, I see so. I think this is not only true for obserability, but with many areas and products where you have a lot of people that need to use these tools but not on a data day basis, which means they're not experts. So right, if I look at at our product and our competitors product, they're extremely powerful products, but most people only interact with let's say the data that comes out in case there's a problem. So maybe this happens once a week, maybe this happens once a month.
If you only interact with it two once per month, it's going to be very challenging. You're not going to be efficient. So one way how AI can help you. That's also what we've built into the product, and I believe our competition is stand that as well, or other players in the space, is using a generative AI to say, you know, explain that data to me, or create a dashboard for this particular incident so that I understand what's going on for the ara of the code that I'm responsible for.
So using generative AI to really become more efficient with a complex tool with complex data when you don't work with it on a data day basis, when you're
basically not an expert. And I think that's where it's huge help. And that's the same with with code generation just on any type of business code, right because you don't write that maybe certain basic algorithms you know every day and I can either then look it up if I need a special sort algorithm on stake go go flow and then copy it in or just say, hey, cope a lot, I need this. Help please set this array in the
most efficient way. And that's a cool thing, and and I see that this is there Another thing that I in the kind of my opinion in one area in the opstability space is standardization. So we have open telemetry as a standard to collect data, and the industry is kind of trying to figure out a way how can we not only standardize the way we collect data, but how can we also standardize the way we analyze the data that is collected, So like, shall we shall we, you know, come up with a
with a standard query language, Shall we come up with something? And I believe that these generated VIIs actually have the power to maybe not solve it, but to mitigate this problem because I can say today, I'm using two X and please Copilot give me that data in two legs tomorrow and to move into two Y, and I can still in natural language say give me that data. And I don't know what the language is that they're using to query this data, but just give it to me, because you are the expert,
right cool the expert. So I think it could also solve this problem. It's interesting you brought up data. I feel like I've been hearing forever, like we have to save all of our data. It used to be for testing, and so companies piled on you know, magnetic tape backups of whatever worthless thing they had, and then it's moved to user analytics and maybe business analytics, and I still feel like every company I worked for didn't know what
they were doing with it. And now it's maybe we can write some mL on top of it. Best LLM model to I don't know generate stuff. Do you think that's changing, Like, are there still like a majority of the data that companies are saving just isn't worth anything and now it's changing, or they're still finding the important aspect in whatever you're saving. There is a
very tough question to answer. I think we are still probably saving on the one set, too much data and it is also not the welst fructured, which means the only way to make sense with this is actually to come up with algorithms and using I guess email systems to to kind of make sense of
this data. But on the other side, I believe we have the opportunity now to educate people, first of all, to come up to use standards, to come up with with with with good practices on unstructured logging, on when they use traces, what should be on a trace, when when you're using when you're creating metrics, you know what metrics makes sense and how should they what dimension should they have, and also come up with with some best
practices. On the other side, what we do, at least from the company I work for, right we we we try to connect the data because what we've seen historically many organizations in the divisibility space started somewhere, right,
we ad Din a trace. We started with tracing, so we've been doing tracing for the last twenty years, and then we added you know, metrics and logs and relyuse the monitoring others, you know, I'm sure like Splunk for instance, right, they started from logs and then they expanded into other
areas. And I think many vendors out there right now, and especially if you, let's say do it yourself and you put in you create your own opstability platform, you have data silos where you have your logs, your metrics, your traces, your events and so on. And this actually makes it
very hard to then analyze the data and put the context. And so one of the things that we at least try to solve is as we interest the data we connected, we enrich it with topology information, so we know that this log comes from this trace, from this service, from this Kubernators cluster, from this end user. And with that we are also more efficient when we run our AI over the data, when we do analytics. And as
I said, I can only speak what we are doing. Maybe others are doing it as well, but I think the problem that we should we need to solve as an industry is don't just collect data because you can, but think about when you collect the data, what type of data is it, what's the meaning of it, how does it connect to other data? What are you expectations, what do we expect this data to be Because you're talking
about data observability. If you have a let's say, a percentage number that should be between zero and one hundred, and if all of a sudden five hundred comes in, there's something wrong. And I'm sure this is the same with with with with other things too. So you want to you want to make sure that the data is clean, that the data is accurate, and that you only collect data that actually really makes sense in the end, because then we can we can also make it easier to analyze the data in the
end. Having a plan before you starts a good idea, it's always a good idea, exactly. Yeah. And to me that highlights like one of the big advantages of using a you know, not only a performance tool like kind of traise, but having organizations like to see INCF that can because there's a lot of work in making all those decisions right and you have to have a ton of expertise in that area to even know what the right questions are
to be asking, much less even get to an answer. And so that's where I think having those resources available to someone like myself who's just trying to write application code to run my business actually helps because then I can use your expertise to solve problems that I didn't even know I was going to have.
And I think that's a hard mental shift for some people, especially younger people, like oh, I can just write this myself and then I don't have to use this product or I don't have to sign up for the subscription service. But there are some long term benefits to, you know, using the product created by a team of a thousand engineers who are solving problems and I'm not even aware of. I just want to add one comment. It's not
only the young people that think they know everything better. It's also some old people. That's their point, and I include myself sometimes, right, sometimes you just think you have seen this before and you know it much better. But yeah, but yeah, right, yeah, you know the scene CF and especially if you think that this happened. I mean, I'm very happy about open Telemetry already obviously, right, it's in the divisibility space and how
everybody's collaborating here. All the big vendors are collaborating, bringing in their expertise, contributing the different types of agents for instrumenting application code automatically and then en reaching a bit open telemetry or we have launched two years ago on our almost open feature to standardize feature flagging, and a lot of companies you know,
are contributing to it. It's really great to see how what the scene CF does because it breaks down a lot of these barriers and walls that you normally have between competing usations for sure. Yeah, because that's a definite change. You know, we've been all three of us have been in this industry long enough to know that that was not how this industry started, you know, like back in the late nineties early two thousands, to think that Microsoft was
going to do something collaborative with another company was just completely unheard of. Yeah, yeah, here we are. Yeah. We always make the comment, I think, Brian, he always makes the comment, this is no longer the Microsoft that I that I knew for sure. Yeah, it's amazing. Yeah, I mean it's interesting you bring up the collaboration. I see.
I see. One of the things that comes up is that a lot of these tools are still incredibly expensive for our small scale and I see that being a reason that especially individual contributors if they're just trying something out or building,
want to even or motive to do it themselves. Actually, it's interesting, we actually have a similar problem at Offerers that we're trying to solve just on the metrics side, and it's just such a small amount of metrics that we want to collect from a volume standpoint, but high number of dimensions, and it's difficult to find something that purpose fits what we're looking for without jumping into a giant product that has a lot of different bells and whistles that go in
a lot of different directions. And the cloud provider we're utilizing right now just
fails in that regard, which makes it a real struggle. Well, I mean this is the reason why you know as well as you say, whether there are our organizations that have thousands of engineers and that we resolve into tough problems like high codinality, lots of data, lots of metrics, lots of dimensions, but as you said, not one hundred percent of organizations have this problem right away, and therefore for a long time it's completely perfectly fine and
to to to use solutions that are free in that open source. But at some point there's a reason why commercial companies exists, and that invests like we we have more than a thousand engineers worldwide. You know, it's not we're not just sitting there and doing nothing and just collecting the checks. We're actually we solved some of these challenging problems that some we many of our customers have.
Yeah, I mean, for sure a lot trying to make sure that I can even a container you run of an open source project inside your cluster on a machine is incredibly difficult to get a high reliability out of it and configured in an optimal way for your environment. It's not a trivial challenge.
And maybe that's one one comment to this, and I know we are probably already way over time, but I wanted to add one more thing, coming back to the EI question that you had earlier, because what you just said is it's very hard to configure and find know how to correctly configure your kubernat Is nodes, your pots, your resource limits, your request limits, and again, this is also something where I believe where EI can help us or
whatever whatever you want to call it, right is basically using the best practices and things we've learned from other systems and then applied to workloads we now see.
So we were working with a partner company in Italy. Let's just give them a quick shout out because maybe somebody's interested in Kamas and they are doing they're running experiments AI rhythmic experiments where they are finding the optimal setting for your JVMs, your CLRs, for your containers, for your kubernatores clusters, and they look at obserability data and then you give them a goal. You said, hey, I want to optimize memory and CPU, but I also don't
want to jeopardize my performance. So you can set them goals and then they find the right configuration based on some machine left in that they know that some
models that they build, and they have some really interesting numbers. And I'm not an expert in Kubernetes down to every single configuration line, but this is where again I would go to at least AI to support me and say this is what I would recommend because I've seen, I've analyzed thousands of systems and therefore give you this recommendation if I CEO system and then I can decide whether
I want to go with this recommendation or not. Makes sense. Yeah, And on the other side of that, I find one of the common things I do with AI is whenever I come across a configuration or some code that someone else has written, I'll just paste it over into AI and say, can you explain what this does? And obviously what is in the end means there will definitely be certain jobs that will be less in demand because they will
definitely be replaced by AI. But I think this also gives us much better opportunity to think about instead of doing the same mundane thing all over, I hopefully have now more time to think about how I can contribute my brain power to something that an AI cannot solve. Right. Yeah, if your career passion is creating security groups inside an AWS VPC, AI might be a career
limterter for you. But if you would just prefer to move on with something a little more fun, I don't think AI is at risk of taking your job away. Yeah. I think continual learning as always a requirement as technology evolves. I've seen AI disrupt more companies though than individual jobs in a lot of areas. So you know, like giant companies that, like say stack overflow disrupt it. But as far as individuals and their abilities like that's still
an open question as far as what that will happen. All right, should we move on and do some picks. Let's do it, Warren, I'm going to put you on the spot. Have you got a pick for us this week? Yeah? You know, I have to have to spend all week thinking about that part time job. Yeah, that's that's actually more challenging than doing the preparation for the episode. Honestly. Uh, this one I think I figured out though. In a couple of weeks, I'll actually be
in Dressden for the Decompiled conference in Germany. I'm actually getting a talk there on how AT offers we iteratively added security as as we grew both our technology product and our organization, and I'm sharing that with who's ever else there. But there's a lot of interesting talks for the one day conference that's there, So shout out to Decompile and I'm sure I'll bring it up again before I actually have to hop over for that. Right on, Andy you were pretty
excited about doing picks. What have you got for us? I actually got two for you. Oh absolutely, So first first, the fun thing. So I think it's very important in life. You know, we spend a lot of time in our job, even though we love it and we're passionates, a lot of time. I think you still need to have something outside of your regular work that kind of keeps you, you know, gets your mind of mind. Is salsa dancing. And I know there's a lot of
salsa dancers, so at least you know dancers in our industry. So if you have not yet tried salsa, you should. And the good news is in any city where I've traveled, and I've traveled quite a bit, there's typically always a salsa community. And salsa communities have been around for a little while, so most of them still organize themselves on Facebook. So if you are kind of post Facebook generation, because you're much younger than we are,
then this might still be a good thing to go on. And then you just say, salsa in Boston, Salsa in Los Angeles. You've found it. So that's a just I love it. When I travel, I go dancing and it keeps my mind off. It also is and it's an exercise, so it keeps me fit and healthy as well. Right, and then the second thing is just a it's just a little tip, not if people
are watching. Still, but on the weekend I went ski hiking, so like not not downhill skiing, but you know, I think it's not cost quanty, but you actually go up on the high hills on the peaks ski touring. Yeah, and unfortunately we have a lot of it's like too warm right now, so that means we had to get very high up to actually get some snow. And then it was so warm and I thought I can just be cool and I just you know, go with my short sleeves and
my shorts and nothing will ever happen. Well, my wife disapproves, she would like seeing me. When I was I was like I made a wrong step and I was sliding down a couple only a couple of meters, but the on the ice and the snow, I have a couple of bruises. And so folks, if you think even if it's a warm day and you think you have no problem on skis because you can you will never drip or or sleep, don't do it. Just wear something long slip. It's just
better. It's painful out of us. It's been painful for days right on. So I have two picks this week as well. If you've been listening to the last few episodes, you won't be surprised at my first pick, and that is for platform Con coming up here this spring. Platform Con is great. It's a five day, free virtual event on platform engineering and there's
just tons and tons of great speakers lined up. So if you if you heard what we were talking about on platform engineering today and you're interested and want to figure out how that can actually make your life easier, make your time, your team's life easier, make your company more successful. They'll at least be a few talks to check out on platform Con. And then at the end I'm doing a live Q and A with some of the conference speakers, so that's going to be fun. And then my second pick is for Obie
Vincent. Obi is a he's a personal trainer from the UK, but he has a great website and focuses on a lot of kettlebell type workouts and I'm a huge fan of kettlebells. As of this last year, I've been working with kettlebells exclusively for about the last year, and he's got some workouts and some guides and some tips and stuff on his website Obie Vincent dot com on how to effectively use the kettlebell and get a great workout in with those.
So, if you've never done kettlebells or you're looking to try to get into working out, kettlebells are just super cool because it's a low barrier to entry and it's dynamic. And actually I've noticed in the last year of using kettlebells. You know, I'm fifty three now, but I've actually gotten stronger by stopping using barbells for working out and using kettlebells just because it tends to line up with how our bodies want to move anyway, which translates to greater strength
improvements. So, yeah, go check out Obi Vincent dot com. Do you have some in that room there that you want to show off? No, they're actually out in the shop, but I can bring some in for the for next week's episode, for sure. Yeah. Absolutely, Yeah, I've got I've got about five hundred pounds of kettlebells and different sizes. Yeah, which that's definitely not needed. That's yeah, yeah, yeah, don't don't do that. And god, I've got to finance this. Now you
can. You can just grab one kettlebell and start with their So that's like the upper If you're you're hitting that number, you know, maybe you need to see over right, or maybe like talk to your therapist about what you're really not focusing on. But that's a subject for a different podcast. Awesome, Andy, thank you so much man. This has been a great conversation. I really enjoyed having you on the show. Thanks for having me. And yeah, yeah, as I said, you know, seventeen more years,
seventeen more years, so now we'll be back. I'll put it on the calendar. Yeah for sure, Warren, thanks for joining me. Love having you as my co host. It's been a blast and look forward to many many future episodes with you and to all of our listeners, thank you for listening, and be sure and check out the Pure Performance podcast as well if you aren't already. And we will see you all next week.