Welcome to Software Testing Unleashed, the podcast for testers developers and all software people who want to create best quality software. My name is Richie. I'm a software-quality coach author and keynote speaker And happy to bring you an episode live from The Hustaf Conference in Budapest. Hustef is one of the greatest conferences I've ever been to. Top quality talks, amazing people and that unique mix of friendly and intense atmosphere you only find at truly great events!
If you have ever get a chance go there seriously You will love it. My guest today is Leandro Melendez – you better know him as Senor Performio. Beside his deep knowledge about performance testing, he is also a very active part in the software-testing community. Just to mention this YouTube channel and his live sessions on LinkedIn... And when I recorded other episodes at Hooster he had his podcast studio next to mine.
so i'm always doing good interviews with lots of people from the software community So please join these channels too. but Before doing that, listen to this episode because we talked about the evolution of performance testing in the last twenty years. What changed? what is still equal which tools are used and how to start a performance test strategy? And now enjoy I'm super excited and for a change, am on the other side of the mic. Super happy to be here! And now you are here?
We're at Hustaf... It's my first Hustef... How about you? My first as well, super happy about this experience…. Ah okay..I think your part of the family is in series. yeah No no no i just uh....i feel like we people blend well in these conference. it feels very familiar friendly. I'm liking it a lot. Yeah, yeah and that's great because you have the studio next to me. so we are doing both interviews. You're doing more lives on LinkedIn. So you all check out Leandro's page. We will link it.
But now you are my guest And i can answer...I can question everything I want to know. Don't be afraid! Be very afraid Because your nickname is in your performance. It leads me into the thing of performance testing. That's why in English performer people thought for some time that it was just like a stage performance, good at dancing or something. I'm very bad Mexican on this level But you can dance? So maybe... If you ask my people! I am a very bad dancer Here in Europe.
i think i mean different levels but salsa..i don't know. But now its performance testing. Nowadays i do performances of stages and talks but still very much aligned on the performance testing practice, now getting a little bit more on observability monitoring. But being in QA trade you start to learn. I mean... You need some manual testing can do functional automation done integration and it's Great that after a while you start to get skills.
Being here in conferences, You hear the other speakers and learn about new tools not necessarily on performance. Choose a little hidden story. I was anti-agile before i started to come to conferences And Here I learned how to embrace it. So That's another one where...you start to grow. you start To Learn More Skills. Now I really like agile. Yeah, i get mad when people don't get well into it.
yeah so It's sets of skills that you can add up and I've been doing some agile speaking performance QA IT. So whatever you need them in salsa dancing will do my best. Okay we'll do the dancing stuff things later on but let's go into it. The Performance Testing because I did performance testing In one project Fifteen years ago and I think it had been evolved since then. And maybe you can give us a brief Introduction about the history of performance testing.
so well up to that moment where we are now. So once upon a time No, at least when i started because uh. I started on performance testing. two thousand seven two thousand eight? I jumped on. the load runner was. That was my first love my first tool. and You're right, on those days it was a lot of load testing. If you have heard me and know I had this common rant performance test is not the same as low-testing but in that case do mention performance? It's a load test!
Yes because we were at Waterfall bare metal servers in the basement or under a desk. I knew many productive servers that were on their desk and person would sit there every day. And yes, we did performance testing in a very different way. In the past. it was lots of scripting from browsers reverse engineering trying to figure out The tokens session IDs correlations. It was madness but i think I call it a Stockholm syndrome because he was fond madness after awhile and you enjoyed.
It's, i love to be correlating on finding where is the missing needle in the haystack. but You cannot be chasing needles. sprint by spring nowadays with agile With the cloud Kubernetes ephemeral environments that will pop up and destroy completely Is not the same game? And I am a strong advocate of moving away from the old practices. You hinted at it very well like fifteen years ago, yes he was there again and that's how you could test your system. but nowadays we also have more APIs.
everything is service oriented...you can test the performance of our systems first of all manually what? That used to be like, yeah. They would burn you in green wood. if you said that in the past What are you gonna have? a bunch of people inside? Of our room clicking at the same time? on those days world was kind of. now You can. it's their user. we could have a Bunch of users release techniques that and In the past they didn't have like.
Maybe they had a few but no blue-green Canaries AB beta testers, like you can have small load tests with real users. And what's better than a real user to do the most realistic clicks and activity? Or directly call those APIs! Stop recording so much from a browser... ...and think of some results…. What are you getting out of performance tests that we're doing?... Again I'm being very careful for performance tests or load tests because it is something big that has changed.
In the past, you had your bare metal server that you have to compile and do their release. And once in a lifetime or every six months of year You had to be ready That single box would take the load Or respond. well at the very least. But nowadays we're already production. If you are doing agile in right way or kind-of correct way Which means for me Atleast one month in production, not pre-productive.
Rollback have the capacity to roll back easily and pay attention what happens in production when once it hits there don't only operations or only SRE. QA is also responsible. so if you have those capacities You Don't Have To Test For Load For Capacity Every Sprint Every Time That You Do A Release And that's a silly approach. nowadays We are generating the load from cloud devices and we're having our system in Cloud infrastructure.
So, cloud to cloud doing a massive test You're just spending a lot of cloud money. Yeah Doing that every sprint yeah And especially because we have the MVP hopefully already production you should use observability monitoring To know That your system is running well? Keeps running well. Your car has been running for I don't know how long have you had your car? Years, maybe.
yeah if You change a tire and you don't race it to a racetrack too add to city capacity No Just check If still works Well feels well with the new tire And i think that's The modern principle For performance not so much. And maybe I'm right or not, so you tell me. Nowadays we have also. it's less effort to scale a system. It is just the volume that you turn on in the cloud and informatimes. You had buy new stuff To increase the capacity of this system. Nowadays its even automatic. Yeah!
You've got elastic systems That will grow based on need which is dangerous as well. because going back to the car analogy I love car analogies, i'm not a car fan but they help me explain. But if your car let's say it's elastic and you can always put more fuel on it If you need it... ...but if has no hole in tank bad performance its not tuned And gives one kilometer per liter. That's not good. Even if you still can't pump more.
so, You got to be careful as well that another tier that nowadays we have too take into account. it may Be responsive and quick and be able to take all the users but maybe your spending Ten times more than you should because a performance in the back end is not The best. yeah So you have to make sure that? Do don't have those holes in the gas tank or the oil tank or leaking Oil. And that's another tier of modern performance, that organizations have to be very vigilant.
But they are still trying the low tests every sprint which doesn't make sense. I... Are you familiar with Taylor Swift situation and ticket company? Oh no! You're not a Swifty okay?! No i'm not a swifty sorry. So Not long ago there was big concert from Taylor Swift And the ticket company that I don't like to name them because they didn't do load testing. Yeah, and They had lots of problems with those situations. It was in the news it didn't made it to Congress and almost just split the Company.
and what's big a big situation? Yes load testing is still important but he's not part of the continuous release. yeah I mean, we all wish that Taylor Swift came to our project every sprint. But no! That's not gonna happen. We are NOT going have Black Friday every sprint. Why are you testing for black friday? Every sprint. why are your testing for capacity? You have your MVP and production. You're releasing little things Measured for that. When a big situation is coming.
Yes, do a big low test but outside of the pipeline Outside of the continuous. That's not a continuous event and that's something that many organizations are not aligned Are not picking up in The best ways, and I think that's what i have been preaching the most uh over the last few years. now You were mentioning the cloud and we had more elastic environments. Beware low testing. cloud environments can get super expensive so be careful with that!
Yeah But that as well goes aligned with, in the beginning. The cloud was cheaper because they wanted everybody to jump into the clouds. so the cloud providers were like yeah this is early promotion like cheaper just getting our platform will look at a long-term price eventually. I have a feeling that it's happening with AI nowadays and everyone attacking the AI adoption in a way that right now is super cheap. and tokens, but it's going to get expensive if we are not using AI in performant ways.
I'm starting kind of call it performance AI especially money wise because you're gonna be worried about... AI may respond quickly maybe really powerful? The general models. But if you have a specific model, let's say you train one based on your code or results. You are not sending so many tokens with all the context at all times because that context in performance can get really big and fast.
And the token expenses... Right now they're saying it is super cheap but why don't you tell them when their real price comes? Lately, I'm advocating a little bit like be aware of what types of AI is it? A general LLM. Is it a model that you're trained? What type of AI? MCPs are really cool but be careful. Why aren't you connecting the? A to A? i think It's the latest thing That is right now. that can get out of hand pretty quickly if you are not careful.
so some performance elements that Yes, he's responding fast Yeah, but that's not the only thing you should care for. so That is another layer of performance. When I transferred back to performance When it's so easy in the theory to scale up our systems, and you said that can be very expensive. So is there nowadays then informatist more focus on doing really performant things? And deep of this system early stages into design phase unit testing a face too.
think about all these performance after we don't have This this need edge off scaling them so much or two. thing about performing in the beginning of the project and not when doing just a test at the end. I'm seeing it more now that teams are starting to be careful with cloud expenses, they're from the begining like oh how can we make this do nothing so expensive?
Because uh... In past some organizations were kind-of careful because there was limited by their size on the box For management, it's always money. If the bill starts to get a bit high like hey guys what is happening? Why are we... why is our cloud environment being so expensive? just stay on. and if you're not careful. in the past people used think performances lots of users all of them slamming their system utilizing.
but nowadays again if your turn on your car You don't drive It's just sitting there on its consuming gas. Same happens with the cloud, if you are not careful tuning it in a right way... If the instance is not killed when it isn't being used. new measurements of performance. how fast? Is a new instance coming up? The new pod or whatever infrastructure we're using? How fast does this come for that new load happening? That's a new set-up measurement.
Not everybody can measure In performance because in the end was just response time from the client and that's it. But now response time for the pod. Yeah, how long does he take? For the pot to be out Once we are not using it? is it like super active? I'm saving as much as possible. i'm dying right away. Oh That has some impacts. okay let's leave it two hours on. oh that also has impact.
so And that's another new challenge by getting into that sweet spot where you get there least cloud expenses, but the best response to sudden increases in loads or certain activities starting suddenly. That as well is if your port isn't on. when you're first one of the platform today You will be the ones seeing spinning wheel for a little bit and like this really slow And everybody after that would be like no it's fairly fast!
Do we care about those single guys who are beginning part bad experience? Lots of decisions, so it's really interesting nowadays the performance challenges and things that you have to pay attention. When I'm now a tester in the development team and said okay we have to think about performance what is a typical way to start on these days when let's go through performance? A person that wants to start their career? No, no. In the project I think as a tester we have to think about performance.
maybe developers should be aware of how do we start and can go into it. So best way is called day-minus one tasks. where you haven't even started your projects. You start setting ground rules. what are going use to monitor if the developers have to create automations to measure performance? or are they gonna instrument, hopefully both? What platform do the platforms tell us their performance metrics. All of those decisions... ...are best practices before you start a project.
But being realistic most projects already running and no one paid attention to these details. And I would say that the best practice nowadays is not focusing on automations but to know the performance and you don't need an automation to know. In the past, that was a common practice because The systems were sealed only the admin had access And it was just like... those where other times Nowadays You can put agents Instrument. there are open source projects.
That if the team adopts them right away at the beginning of project. or okay? You have been already running for a few years with this product. Add it right now, start building whatever you can. Have in place with the cars again I like that example. You don't have to get off your car leave the hood pull a little thingy check the oil That we used to do? Yeah! You already had everything in a dashboard. would you buy? A car doesn't have a dashboard that tells you Everything. nowadays It's the same.
Would you start a project where Do not have a Dashboard To tell you all The performance metrics of Your system In a way that all the team will understand it. You'll understand most of things on your car dashboard like speed, gas and good ranges numbers not just a voltage from a sensor. No I want to see something human with numbers. colors make it pretty. So i think those would be first steps. have observability agents and telemetry so everybody in the team We'll be aware of that.
Yeah, and then let developers make it as easy as possible for them to create automations and measurements before after during. There's a shift left shift right Shift up and shift down which not everybody talks about. have observability at the browser level At the backend level infrastructure level. so It's many things that you could do but the bare minimum have observability, know what is your performance without doing anything. Just know it!
Okay I think a great insight to start on and begin with all this topic because its such huge complex so you need to start at any point. sometimes organisations get lost. sadly the sponsors here might not like me too much but... many times it's like the tool. What tool am I gonna get? There is no A-Tool and i always get this question, nowadays...I used to dislike it but now I just smile. and what's the best tool for performance? And if I give you a dinner table.
have all that silverware..what tool would you choose for eating? Wait! What?! A single tool?? That's a point. You need to embrace many tools, many platforms. Whichever works the best for you. try to have it in place and don't lock yourself inside of A single type of tool. that super important. To start with performance Yeah And the vendors always have this pitch spiel that oh low testing low testing because I sell at all. that is no testing. so you should No, no, no know.
and that sold fifteen years ago as you say Even training, like some of those practice really old. Great! Thank you very much Leandro for these insights in the beginning into performance thinking. it's also a very new topic again from me to think about this stuff. I appreciate that very much that your joined the show here and doing a lot for the community and have your own channel on YouTube, we will put all of our links in the show notes. And you must join him!
I can subscribe, activate the bell... Thank you very much, I wish you great conference and good interviews here. See ya soon! And I look forward to more opportunities. Thank you so much for having me, it's been a pleasure!