Hello and welcome to a new episode of the podcast software testing. Today I have brought a basic topic with me, and it is about test design methods or test design techniques. One who wrote this as a passion for the flag, who knows them all and has penetrated deeply, is Rick Marcellis. With him I talked about why test design methods are so important. Why test design methods are so rarely used and which one is best to start with.
And as a little goodie, Rick also gives out some copies of his books, which you can win here in the podcast. You can find all the information in the show notes. And now have fun with the episode. Hi Rick, nice to have you on the podcast. Yeah, Hi Richie, good to be here. Thanks for the invitation. Yes, thank you very much. We are here on the testing retreat, the 28th testing retreat. Yeah, in a very nice venue in Belgium.
Yes, yes, it's very, very nice here. So it's now my mobile studio here. And we talked in upfront, what topic is your passion? What do you like most in all this testing and quality stuff? And you said test design techniques. Yeah. And I said, oh, yeah, we have to do an episode about this. Yeah. So what is your opinion? We are learning test design techniques in foundation level and so since 20 years. Does it work in reality?
Well, I have been teaching test design techniques for almost 25 years now. And my, you might say my frustration is that many testers still don't apply them. Yeah. So the problem I see is on the one hand that nobody requires them to use test design techniques. And often when quality of IT systems is quite bad, you don't really need them because if you push a button, you already find the problem.
But if you have good quality or you want to really give a convincing answer about the quality, test design techniques are really useful to show what kind of coverage you have. But one thing we always say in TMAP is that you should always combine experience based techniques, for example, exploratory testing in combination with test design techniques. So with the test design technique, you can prove that you have covered everything that is important related to your risk level.
So higher risk, take a more thorough test design technique. But also you want to have confidence that things work as expected. And often confidence comes by a combination of coverage and experience. So the test design techniques, do you think there's also a lack in knowledge by the testers or is it?
Yeah, well, you see, ISTQB, if you take ISTQB foundation and the advanced level test analyst and the advanced level technical test analyst, all in all, I guess you learn somewhere between 15 and 20 test design techniques. And basically that's part of the problem because you learn way too many. And about 10 years ago, my colleague Bert and I did a research about test design techniques and we found that there are about 25 to 30 test design techniques. And why don't we know the exact number?
Well, because for some techniques you can debate whether they are separate techniques or just a variant of the same technique. So we'd say 25 to 30, anyway, many more than anyone will ever apply. And also, so I have with me the latest TMAP book quality for DevOps teams and TMAP exists for 29 years now. And in earlier TMAP certifications, you had to show proficiency in 19 test design techniques and approaches, which basically is not possible in a three-day training course.
So the only thing you learn is how to check the right A, B, C or D on the exam and you don't really learn. So when we introduced a new TMAP certification scheme, we decided that on the course you start with, quality for cross-functional teams, you get five test design techniques and one approach and that's your basic skill set. But we spent some more time and also some more exercises on it so that you actually are able to use it at, let's say, a beginner's level.
And then of course you need to get better. And one of the reasons why I think that many people don't apply test design techniques is simply they have not come across the hurdle of learning the test design technique properly. Because on the exam you just know the basics and you can answer the exam question, but in real life it's always more complex than on an exam. One of the things that Bert and I discovered in our research is that it's quite easy to group test design techniques.
And what we did, and I'll show it on the camera, well, people that only listen to the podcast won't be able to see that, but I'll describe it. What we found is that all test design techniques can be put in four groups. And the four groups are process-oriented test design techniques, for example, path testing where you have several levels of path coverage, or state transition testing where you have state coverage or transition coverage, one switch or zero switch, whatever.
The second group is condition-oriented testing, which is, for example, decision tables where you have various conditions and they determine the test cases. Third group is data-oriented, for example, equivalence partitioning where you have a data item and you have several partitions. And the last group is what we call appearance, and that is how the system shows itself to the outside world. And on one hand, that is things like syntactic testing.
So, are the buttons on the right spot? Do we have the right font type, the right colors, the whole style guide? And on the other hand, appearance is also about the non-functionals like performance, usability, whatever. And by knowing these four groups, a good thing that a tester can do is make sure that you know at least one or two techniques per group.
Because if you know all techniques from only one group, for example, I did coaching in a team and they decided to use the so-called elementary comparison test. That is one of the most sophisticated techniques. Also, some people find it a relatively hard technique, but if you use the proper templates, it's not all that difficult. But it's a condition-oriented technique and they decided to always use it because it's a technique with high efficiency and high effectiveness.
So, that sounds great. And then they said, "Yeah, but it doesn't work for us." And then we determined they had a data-oriented problem. And then a condition-oriented technique is not the right choice. So, do these five techniques you teach, you selected, represent these four conditions? Yeah. So, we have five techniques. So, we have equivalence partitioning and boundary value analysis from the data group.
We have path testing from the process group, decision table from the condition group, and syntactical testing from the appearance group. And additionally, we have exploratory testing as an experience-based approach. And then in the T-MAP scheme, we have a second training course called high-performance quality engineering, which again has eight techniques per group so that people that have done both training courses have about 10 techniques and two approaches.
And basically, that is the toolbox that 90% of the testers will need during their career. Yeah. When I come to clients and to test teams, what do you think – how is the transfer of these techniques in the daily work? Do they have to sit with pen and paper and do the equivalence partitioning and the boundary value or using some tools?
Or some testers always say, "Yeah, I understand the idea. I do it more intuitive and decide the boundary values may be more…" Well, of course, it depends a bit per technique. For example, the boundary value analysis is the most intuitive technique. Even people that have never heard of the word test design technique often already do a boundary value analysis. But for other techniques, we created templates.
Because one of my colleague teachers of T-MAP, who I know for about 20 years – well, over 15 at least – she admitted that teaching people decision table testing was always quite hard. And with the new certification scheme, we also created a template that we also put on the T-MAP website, so anybody in the world can download it. And since we have the template, teaching decision tables has become much easier because you don't have to teach how to make the table.
Because you already have a template. You simply say, "This is what the table looks like. You don't have to learn how to make it. And then this is what you fill in here and there." So, you put in conditions, you put in actions, and then you determine based on the true/false combinations what is the expected outcome. And of course, the expected outcome is the thing you basically want to have. That's the core of test cases.
And how do you teach the tester to learn how to choose which design technique is appropriate for the problem I have? Yeah. Yeah, well, so we make them aware of the fact that basically there are four things relevant, also described in the book, for selecting a test design technique. One of them is the kind of testing problem you have. So, which group do we select from? Another one is what kind of quality characteristic is it that we are addressing?
For example, if it's performance, well, then you quickly come in the group of appearance. The third is risk. So, high risk, we need more coverage. And we basically say, if you have three risk levels, that's usually enough. Why? Because some people like to have a percentage, and then I always challenge them. Is 72%? Is it more than 69? So, we have just low, medium, high, because with the test design techniques, generally, you also can't have more than three choices of coverage.
But then based on the risk level, you decide more coverage or less coverage. And the last one is also an important one, and that's the skills of the involved people. Because sometimes they don't know about the test design technique, and then you have two choices. You can say, let's select another technique that they do know, or let's educate them. And currently, I'm in a project where the business analysts are supposed to test the functional acceptance testing.
So, they test the business processes, and they have quite good process flows. So, I was actually quite enthusiastic, because not always you find good process flows, but they have good process flows. So, I told them, there's a test design technique, path testing, also known as process cycle test, that is very well fitted for this. Will you allow me to teach you? And they were enthusiastic, because sometimes people don't want to be taught techniques, and they think it's too much trouble.
But those were enthusiastic. So, we sat down for like an hour and a half. I explained it. We did an exercise, and they said, oh, yeah, this might work. And last week, we did a two-hour session with a live example, so their first process flow, and we created the test cases. And of course, we have a template for that. So, it's just a step-by-step approach. And if you follow the steps, you will end up with your test cases, and then you prove the coverage.
And then, of course, they came, yeah, but if we have those test cases, then we have a situation here that we don't test. And I said, well, but do you cover all these paths? Well, yeah, we have covered all of them, but this is a specific situation. And then I said, well, this is where coverage-based testing meets experience-based testing. So, please go ahead and add an extra test case based on your experience, but don't throw away another one because you need your coverage. Yeah, that's true.
I like very much the approach to learn the techniques with real-life examples because in some courses or seminars, you learn it, the state transition with an ATM, but you don't have to test an ATM every day. And the transfer to the daily work is for a lot of testers not often very easy if they don't see how it works in their environment, in their domain. So, I like this approach to do it and to see so that these testers can see how it works in their domain.
How do you – or do you have experiences with these techniques on the more developer-near testing, so unit testing and so on? Because sometimes developers or architects, they say to me, all these techniques are stuff for the testers and not for me, so I'm doing my own thing here. So, what is your experience? Well, developers, especially in unit testing, they often focus on code coverage.
And what we have described – and the Quality for DevOps Team's book has four main authors and two of them are developers. So, two are from quality and testing, the other two are developers. And with them, we looked into code coverage and the most simple coverage measure – because often developers only say, "We have 100% coverage." And then it doesn't say much because you first have to specify what kind of coverage. And often they have line coverage.
Now, some developers manage to put five statements on one line. If you execute only one statement, you already have 100% coverage of that line. So, I always say, at least don't do line coverage, but at least have statement coverage so that every statement has been tested. But better even is decision coverage because the most important statements in your software are the decisions. Because a move statement that moves some value to a variable, well, the chance that that works is quite high.
So, of course, you can test it, but there's not much of a risk there. But in the if statements, the decisions, or while loops, and you want to know if it does the loop the number of times it's supposed to do, then you need to have decision coverage because you want to have the good situation and the fault situation. And to manage to have decision coverage, you can already apply simple techniques.
For example, equivalence partitioning helps with finding the wrong result and the right result because you have two classes and then you must make sure that you have a test case for every class. And equivalence partitioning is the most basic, simple test design technique, but many developers, they only test the happy path. So, they forget to test the wrong situation. And that's already how we can help them.
And, of course, equivalence partitioning is almost as intuitive as boundary value analysis. So, it's not too difficult to teach them because it's more a thing of awareness than actual teaching. Because most developers, if you tell them you should not only test the happy path, but also what happens if you do it wrong, they say, "Oh, yeah, yeah, well, it makes sense. Yeah, but then I get twice as many test cases." Yes, you do. Yeah, yeah, yeah, yeah. Correct, yeah.
Do you see any test design techniques which evolve today or are those all test design techniques that are known since 20, 30, 40 years? Yeah, almost, well, let's say, as far as I know, all test design techniques are described in books that are more than 20 years old. You have the first book I know is The Art of Software Testing of Glenford Myers, which is 45 years ago. It describes boundary value analysis, equivalence partitioning, well, some others. Then we have the book of Boris Bitzer.
And we have the old TMAP books because ISTQB often refers to Boris Bitzer and a bit to Myers. And they also refer to some of the previous TMAP books. But one of the things that I'm still puzzled is why not more people have adopted path testing. Because in ISTQB, the only process-oriented technique they teach is state transition testing, which is about states. And not too many systems have states, and especially if you want to test acceptance testing, you want to test flows, business flows.
And path testing is about covering all the paths in your business flow. So, yeah, and I now notice that because I told you I have currently a project where I guide the business analysts in doing path testing, and that actually works for them. Yeah, that's also their view on the system. So, they see the business flows. Yeah, they think about the business process. Yes, yes, yeah.
Rick, do you have one special test design technique you like the most, which is your absolute favorite because it's so sophisticated or so nice to use? Well, I like many of them. So, let me give you my top three. Yeah, okay. So, not surprisingly, path testing is in my top three. I also like decision tables because they are very good for testing each and every possibility. The only downside of decision tables is they quickly grow out of hand.
If you have more than five conditions, it doesn't work anymore. It gets too big. But in most situations, nobody has five conditions to determine one outcome. So, normally, it works fine. And the last one is, I already mentioned it, rather complex, the elementary comparison test. And to properly learn that, I'm convinced that I need to give people a one-day workshop, which I've done in the past. And then they get it good enough to apply it in real life.
And for that, we also have templates because elementary comparison test tests multiple decision points. And for every decision point, the coverage will be modified condition decision coverage. So, why is it somewhat complex? Because first, for every decision point, you have to apply modified condition decision coverage. That gives you what we call the test situations. And then you have to combine the test situations of every decision point into test cases from start to end for a process.
And so, if path testing doesn't give you enough coverage, because multiple combinations of condition outcomes may lead to the same paths, and then in path testing, you only have one test case, and in elementary comparison test, you will do all the possibilities. So, it will give you a little more test cases, but it's the highest coverage you can have. Very strong test cases. Yeah, if it passes those test cases, you have good confidence that the system will work.
Great. Thank you for your top three. So, you mentioned the book, and we have here for the podcast audience a special benefit like in other episodes, we will give away five of these books, so you can win them. You have the link in the show notes. And then good luck for this. And I have this book at home too, and you gave it to me just last year, and I like it very much as a reference for my daily work. So, I can just do the lottery and win this book.
Yeah, Rick, thank you very much for joining this podcast. It was very interesting. I think test design techniques, this is a topic we have to pull up on the stage more. So, I'm happy that you are doing that, and for your work, and for your passion for these techniques. And yeah, thank you very much to be here. Well, it was my pleasure, Richie, and thanks for inviting me. And I look forward to the remainder of this weekend here at the testing retreat. Yes, yes, we'll see what comes.
Thank you very much. Bye. Okay, bye-bye.