Welcome to Software Testing Unleashed. The podcast for software developers, software testers and all people in the software engineering area who wants to create amazing and good quality software. My name is Richie. I'm a software quality coach keynote speaker author and podcaster And my guest today is Yanni Grönmann! I met him first at the Istanbul Software Testing Conference... ...and he gave one speech i clearly remember after all these months.
now He's managing partner at and his mission is turning quality from the cost into competitive advantage. He works with teams, management product people developers and testers to recognize where their product quality comes from And how To make it an asset. Jani has over twenty years experience in The software industry and believes that Quality means business. We talk today about KPIs About key performance indicators And therefore a specialty for quality metrics.
How to balance the matrix, how choose between quality and quantity metrics... ...and overall goal with question what do we want achieve in this KPIs? Now enjoy episode! Great, great to see you again. Yeah it's great that you joined us here. we met in Istanbul at the Istanbul software testing conference and I saw your talk also. i think Jilat mentioned for episode two already on my backlog so now is time. make this episode together. It was a very interesting topic too, so maybe we can do further.
episode two about this topic what you did with AI. But today we have more foundational topics for testing. really good thing I think it's for projects and doing software development is about KPIs and quality metrics. As I remember, it's a huge topic. We can so much measure and do the wrong things... So i think that is very important. to talk about these topics but go into them! Just bring us in on what are KPIs for? And then we will see!
Well, KPX stands for Keep Performance Indicator and it's like a business term for metric that is used to measure some development in an area. For example improving processes or stuff like this. but why its important for us in quality businesses? Is often we are trying make changes in our work, environment with clients etc. I'm a consultant so i talk about my client and that's kind of context. So it is very often the case we want to change something.
There are some kinds of transformational ongoing And teams wants improve their ways of working. Teams wants to improve their lead times quality and testing outcomes. So if you cannot measure what's your change, it is really hard to lead the changes And its' really hard like understand are doing something right or should try something else. so that why we need some metrics. That an interesting topic because many of us have KPIs in use, but those are maybe not so well-placed metrics all the time.
That's what makes it interesting! Yeah, yeah. Because there is such a huge overload of possible metrics we have all over the literature and in their community And so I did book with Harry Sneed and Manfred Baumgart. No it's in German It's software matrixing that this very fat book. A lot of metrics in there but for most people its hard to decide which metrics are really useful especially when you go into software quality.
So when you look at your experience in the project, what are typical metrics? What really brings us forward and it gives information that we can use for our transformation and reaching out quality goals. If I start from a typical one... In testing we have amount of tests And also like unit test coverage percentage Like those are kind of good metrics, but we humans if I have a KPI that my performance is measured by the amount of tests.
I write for in a certain time period It's it's very likely That i start to play that kpi so so that i can get most out Of the bonuses i'm promised and stuff like that. So setting those KPIs Is they need a counter metric. So there should be optimally a pair of metrics, so something that you can show side by side on your dashboards.
For example like mean time to resolve a bug for example or some kind of issue in production Can't be good KPI and but that would be balanced by something like re-open rate. so we could be sure We don't just like close the production issue, issued tickets and that's it. But we also measure. those issues are not coming back. so those kind of pairs or metrics are important. I think this is very important, often when i come to a client and talk with the team.
They have always in their definition of done that they reach about unit test coverage at about eighteen ninety hundred percent or something like that. but don't look on quality for these tests. so it's just amount. as you said if a certain level of amount, writing test cases and test cases until I reach this threshold. So what would be...I think there's typically part with the unit tests because it is so easy to measure. What is therefore counterpart?
To get more quality into unit-test case for example? Well we usually measure like test pass rates. Right, so how green we are? That should be balanced for example with... How much we're missing even though we are green. So a counter metric of could be for test pass rates For example. it would be like escaped defect rates or flaky test rates stuff Like that. We have more control over the metrics. Okay
and When we look for the classical testing job, so writing test cases and finding issues... Do you have more examples of what good metrics are that can be used in our projects? Where I come from is thinking about quality engineering. So shifting left and having continuous feedback loops good reliability metrics, for example. And also on the other side I like to promote product-minded development thinking.
so instead of measuring like testers with different metrics than developers in personal level is not a good thing. So there's this thinking that one team and number type of thing. So from quality engineering, this balanced metrics like the team can own for example that mean time to recovery is good one. and well then again what could it be in practice? That's one T-one number.
For example, if I remember correctly Spotify has this time spent listening metric that they share with their teams and Airbnb have these nights booked or something. Some kind of metrics And so i would take it up a notch like from purely technical level. Of course those Dora metrics are important in the technical sense but to really, like focus the team in some... In the same direction. you should have those kind of a bit more business-like metrics. To share with the team.
and one part of product minded thinking is that you should involve your team also into discussion with end users, discussions with business people so they understand A bit like where the revenue comes and how they can influence that. And let that influence in their work, so quite a convoluted answer there. but... So you mentioned to take more God on higher level of the metrics?
What I hear out there is also to look on the successes of the product in, so gain more listener time or something like that. To feel my work as a team has an impact and it's...I can put some things into this number to increase these numbers. Yeah you did have like agency in your work. So when you're a developer or tester, so we need to have that feeling where the word matters and it's like useful what you do?
And...so are not just producing tests or executing tests but contributing to end quality of product on good customer experience those kind things It was interesting. I did in the German podcast an episode about metrics, but I think two years ago now and there was one guy saying that He just needs to matrix it's how fast can I ship? To production And how happy is the customer? so So it's a very very high abstract and its.
But when we see on this level Is there any value for the metrics we do in the project? How can they help us when I think of defect rates and something like that or coverages. Yeah, Of course! We can go too much into technical level in metrics because we're going to measure anything from all kinds of data sources. So it's also important to keep that kind of simple, so for example don't set more than three or four like KPI metrics per team. It's focused on some direction.
but lead time to production as you mentioned is a good overall metric because And if you have some kind of benchmark, for example that what the typical lead times are. The kind of team that you're working with then You can ask Western Are we reaching a goal? If not why not? and like try to figure it out With the teams at what needs be done so they could reach That lead time goals.
For Example How Can We discuss and decide which metrics are good for the project to view because there are different goals. I think that's one thing, being in a team is better than knowing where we're at with our metrics. but on the other side it's more about reporting to stakeholders or managers so they want to know if their quality is good. how can you build this set of metrics? Problem is that, for example product management usually overworked too.
They don't have much time to discuss these kind of things but you should find a way where they can have their ear and start... I would start by asking what's important like the classical three-way thinking that what's important for you, personally. What is important to our clients and the company producing this stuff? So from there asking more questions about how we reach... Are we reaching out goals or are they where it should be? like just producing features for feature sake?
so why getting to the Y of things? So that's where I would start, but it can be like a long discussion if different sides of the discussion are entrenched in their positions. And i think its often also problem to know what What's meant with the metric? Because metrics can be interpreted in different ways sometimes, so you have to clarify what is really inside this metric and what we learn from it. I think there also a discussion for every metric that has to do... Yeah!
It should be understandable and simple enough balanced by someone who could be reviewed periodically, we measure something and never act on it. It's useless metric. so I think the ownership of the metrics is also very important because if you don't act in your measurements or metrics then no progress may be made. for example that test pass rate If you just ignore the analysis of what this means. disabling tests and making your metrics look better doesn't really increase the quality, does it?
Yeah yeah that's true. And there are some metrics I remember from what I learned when I was a junior tester. And, and I wonder if they are still correct? So one was uh... If we test-and-test in tests We find at the beginning lot of defects. Then this curve gets more flat. The new defect is ... Is it a valid metric? Now that we think about Agile and all the complexity of software development Do you have any opinion on whether or not it's usable? Or is it one of from the nineties?
Yeah, I remember seeing that kind of curves myself too. And to a certain degree yes It's like when you have a mass of tests and those will. if product this would be there would no changes in the product. Those we'll find debugs they found. That's it. There are certainly levels of the effects that those tests can find, and I think you need to have an understanding.
You can develop a set of tests at certain points where actually the counted metrics come into play so also measure how many defects are escaping due to production. So this gives your idea on how good your test-set is Even if it's passing, are we not testing something that leaks defects in the production? So what I didn't know as a junior engineer was you could have these kind of guardrails or counter-metrics. That would help.
and why...I've always been looking at like the amount for coverage percentage ninety percent unit test coverage and then still have defects in production. How can that be? We are perfect! Yeah, we live in a very complex software development world so there's some many impacts on everything. So I think uh...we really need to be clear what we can measure there. And how do you use these measures too? One thing I think of is also when, uh... When i see my clients and they are counting their issues.
They have in production What typically misses? Is that they analyze these issues if this really a software bug. So sometimes there can be so many causes often defecting In production And If we see from the development perspective for us Typically only counts there once, which lead to missing test cases. The bugs in the software may be requirement issues or something like that.
so a good analysis of production issue is very important for getting information back and get better in development testing team. Yeah it's true and kind of art doing good rules because analysis really is an asset on your team in your own toolbox. And you mentioned the requirements, causing defects and that's also something. sometimes at least in testing community all test doesn't acknowledge that faulty requirements can lead to defects or suboptimal test design.
because good requirements or understanding what should be done. What's lacking a bit, I think is that as testers we should reach out more to the requirements engineering side of things and also get an understanding how to test business ideas and like how to challenge the product management. And, on how to reach out them. so we are too much still tucked away in right part of development life cycle. That's sad that we would have a lot to give also for the product.
Yeah, I think that's a very important thing to look at the requirements there and get better on this side too. And understand what is in it? In the beginning you said that... ...you use KPIs for transforming teams so they can make them better. We talked about the KPIs which are more high level to what the successes outside, which defects are going there. And when I'm in a team that is huge... gap between me and this metric I use.
How do you break these information down that can put in a change into the team to transform, get better on it? This is so abstract to be at high level. with my daily work writing test cases or doing automation tests... how do we break them down? Some members of engineers like being distanced from the business goals, they only care about algorithms and coding. And technical side of doing things. that's fine but then there are people interested in why we're building this thing.
so maybe start with those people who naturally are interested to know what these products look like. better and also important things is when you're building the KPIs, You need to bring your team into discussion. And not just like dictate them from above To the team and say that okay we are now measuring your dissent That be prepared for the consequences.
So it's...you need to build them from ground up so that ideally I would like bring a product person into discussion and describe what's the world outside looking like, then have some technical data about performance for example. What are our lead times? What amount of rework is always good starting point. so how many percentage of your time you're spending in reworking something, fixing bugs or figuring out why something doesn't work?
and then maybe ask the question would you like to do something else than fixing bugs for example? so if instead of fifty percent of your time going to fix bugs what it could be? Like ten present sounds good right. So there I'd start building that understanding. that way, we need to build these metrics and use some time. To be able to see where they are going? I think it's very important because this involvement of the team also leads more relationship with the metrics.
If i'm part of a creation or metric more touchy feeling to this metric and I want it better, going better. And i will do my stuff in the context of getting it better here. Yeah that's right! That comes down to the ownership like if you can build a spirit within your team. they care about their metrics and understand what they are about. They're not just numbers, but they measure some real-life thing for example that the customers are caring about. so it's like a win-win situation.
Yes, Jani. Thank you very much for all these insights into metrics. I think it's a topic which can be discussed on all levels and all depths in long time but i think its good overview how to see on metrics And decide what one gets at work For transformation and reaching the quality goals. So thank-you that we are part of this show. I hope we can do another episode in the future, too. And i wish you all the best and thank you very much! Yeah thanks Gritzy for having me on.
it's a huge topic but uh... It is nice to talk about these things and let us see if We Can Do In The Future!