[Best of 2023] #139 - A Developer's Guide to Effective Software Testing - Mauricio Aniche

Hey, a quick message for those of you who are listening to this episode on Spotify. I have a small favor to ask. Spotify now allows mobile users to rate podcasts. I would really appreciate it if you can take a quick pause to go to the Technically Journal podcast page and leave your favorite show your best rating on Spotify. It will help me a lot to get this podcast to reach more people on the platform. Thanks a lot.

Which is a good segue for us to start talking about your book Effective Software Testing. I think one key sentence that I pick when I read your book is that you mentioned to be an effective developer, you must become an effective software tester. But in practical world, sometimes developer and tester, they are two different roles. So tell us more about this. Why do you say that to become an effective software developer, you need to become an effective

software tester? Thanks for the question. I think it's because as a developer, it's your responsibility to make sure that what you do works, and the only way to do this is by writing tests that prove to you and to your team that your code works. So indeed, before and I even worked in environments like this where I would just focus on coding and then I would send my software to another company.

This company would do the testing for me, they would send me a report, they would fix the bug, so on and so forth. And that's just not productive. Back and forth between these two teams is too big. And I think, again, it's our responsibility to make sure that things work. And automated testing is just such an easy and cheap way of doing it. And that's why I say in the book that effective developer is someone that effectively tests

what they do, right? And I think, I mean, I don't know, I was a developer last time. Sometimes developers can be very confident type of person, right? So we rewrite the code, we test a little bit during the local development, we think it works. We always think it works. Many developers probably do not enjoy writing tests for some reasons, maybe apart from unit test sometimes because it's very

close to their workflow. So how would you actually invite more software developers to actually have more passion or energy to start writing tests? If they have the perception that, yeah, I'm super confident of my code, I don't want to spend more time to write automated tests and things like that, yeah, that's a very interesting question. So the first part was you mentioned about developers being confident, right. And what I always say is you never know. You don't know how many times a

day you make a mistake. So you program something wrong. And you only know this once you have tests. Because if you don't have tests, everything that you code looks perfect, right? You just believe it works. As soon as you have tests, then you see, you know, Oh my God, I'm breaking my tests 50 times a day. And then you quickly realize how much you need them because we break stuff all the time. Now, how to convince developers to write tests? That's a very deep question.

I think there are many angles to this. One is to show to them and to have them practice so that they don't see writing tests as something that is a burden. They just get used to it. They get proficient with writing tests, not only with the testing frameworks, but on writing code that is easy to be tested on. You look at a program and the test cases emerge in your head, and all you need to do is to code them in your programming language.

So there's this aspect of just practicing and getting better at tests itself. The second one, if you are in a very large and complex software system, things are too complex, right? And then for you to have this pleasure back in writing tests, you need to make sure it's easy to write tests. And this means as a company or as a team, you have to invest in a small infrastructure that facilitates the act of writing tests.

I was discussing this yesterday, actually, not here at the company, but in another university that I was yesterday. And I was saying, if you're working on a very complex software system, to test something, maybe you need to know to instantiate 10 entities or have data in 30 different

tables in your database. And that's just because your business is complex and you have to have some something in there that makes this easy for you That in a bunch of couple lines of code you can set up the scenario you want to test and then you can do the test. So those are the two perspectives I see. One is again getting good at testing itself and the second one is having the proper platform so that you can write the tests.

Totally makes sense for me. So from my point of view also sometimes we also need to maybe introduce back like what you mentioned in the beginning, the pride, right? So how can you tell that your code works now and also in the future? There's no other way to do it other than having some automated tests that can actually prove that your work is always past, right in terms of test cases.

And I think many developers also think probably writing more code means like doubling their effort, right. So I think this must be changed as well in terms of perspective, so that you don't finish writing the code if you don't have the test. Maybe just some perspective from me, Let's move to the thing that you mentioned in the book, right. So will you mention about effective? And there's another keyword that actually you mentioned in the book which is systematic.

So I find these two things are very interesting the way you explain in the book. Maybe if you can. Explain here as well what will be your advice for people to be a more effective software tester and also to be. Systematic. In coming up with those tests, yeah, indeed. So they make a very strong point that maybe you can be a little bit more systematic when we approach testing. We wrote a paper a couple of years ago, and in this paper we asked developers to write tests.

So we gave them a program and write tests, and we observed them and they were thinking aloud so that we could see how they were coming up with test cases. And we observed a couple of things. Maybe the most important one for this conversation is the developers were never systematic. They were always, you know, just following their hearts.

They would look at the code. This feels like the next thing I want to test, they would go and they would write the test and then they would repeat what is the next thing I want to test. So they would just follow their hearts and follow your experience. And feelings are very important when it comes to testing, but it also opens up space for mistakes. Maybe you're not in a good day

and then you forget something. 2nd, if you're always using all your brain power to come up with test cases, even the basic ones, you're not saving energy to think of complex cases. So when you're more systematic in terms of testing, that means you know you just follow some sort of cake recipe that helps you to come up with a bunch of test cases that are quite easy to see. You can put them on a checklist basically, and then it is your brain power to focus on test

cases. That really requires human smart people thinking about it. And the fun part is there are lots of techniques like this If you look at academic books on testing, even the books from the 80s. Bigger goal back then was to come up with a recipe that teaches you how to write perfect tests, and there are lots of good ideas there and you can think of.

I'm not going to give a tutorial here, of course, but you can think of basic things like, for example, if your method receives a list as an input, there are a bunch of interesting cases that always makes sense. So for example, what happens if the list is empty? What happens if the list is new? If there's no linear programming language, what happens if the list has just one element, right? Because sometimes your algorithm changes if there are multiple

elements or just one. So there are a couple of basic stuff that you don't even have to think and you can write those tests and they are usually good enough to get you to a very high coverage already and then leaving space for you to then focus on boundary testing, you know, corner cases and this type of stuff. And being systematic is something that you see in other engineering fields, right.

And I think I cited in my book, there's this very famous book called The Checklist Manifesto. And there's a story there backed up by research that shows that medical doctors that use a checklist before some surgery, they make less mistakes that medical doctors that don't. And you don't have to be ashamed of following a checklist. It's goods. And I think that's the point of my book when I say we can be a little bit more systematic. Of course we don't have to be

systematic all the time. It's just too expensive, maybe too complex to be systematic all the time. But identify, you know, like I'm testing a complex method here, let me be a little bit more systematic. So that's the whole idea of being systematic when it comes to testing. Yeah, I think your point is right. Sometimes we have typical use cases, just like list or maybe you'll have an API, failure cases, success cases. So all this can be made as a checklist. I've heard about Checklist

Manifesto as well. I think the author named Atul Gawande. And I think becoming systematic in solving this kind of a testing problem is really crucial because in your book you mentioned that if you are systematic, you can assign any developers to the same problem. They will most likely come up with the same test suite, right? How cool is that? Because sometimes we feel that we only need a certain testers to come up with a comprehensive

amount of number of tests. But I think the systematic way is actually to have any developers working on the same problem but they can come up with the same test suite. That I think is really really powerful concept. And I think it also becoming effective means that you write the right test, because I think it never ends right. You can come up with any number of tests as long as you can

produce the test inputs right. And I think to come up with a certain right amount of test is very important. Which brings us to some of the techniques later on. One of the techniques that is commonly, I don't know, discuss in the software world is about. Test pyramid maybe? Let's start from there first. Do you think this is a systematic way to think about producing tests and what? Is your view about? This testing pyramid.

I think the testing pyramid helps you to be pragmatic when writing those tests, because one thing is to come up with the test cases, right? So those are the inputs I want to give to my program and this is the expected output.

The other one is writing this in code and making sure that this works nicely in your, let's say, development process, in your CI and etc. So for example, if you just write end to end tests, maybe your test suite will cost you too much to execute and then at some point this becomes a

bottleneck, right? So I think the testing pyramids gives this pragmatic point of view on, hey, we're going to have to automate this test and we're going to have to maintain those tests and we're going to run them the whole day. And I like the idea of the testing pyramids very much. And the idea is that at the bottom you have unit tests, right? And why is it at the bottom?

Because they are usually cheaper to write, they are cheaper to run, they tend to be more robust, they tend not to really fail, they are not flaky in general. And then you go up the pyramid, you have integration and end to end tests and you still have to do them right. You have to, but you do them a little bit less. Maybe you focus a little bit more in the unit test, maybe it's OK to maybe have some duplicated tests here if you're unsure. Are you covering the same thing

or not. I think it's OK to make this sort of mistakes in the unit test, but it's integration and end to end test. You're going to prioritize a little bit more and I like this idea of the pyramids. Although you know in the software engineering at Google book I think from 2018, I think they bring a new perspective that I also appreciate very much that is forget about unit testing and integration testing, right, just separate your test

suite between. Is this one fast enough that I can actually run it during a pre merge together with your merge request or pull request or is it super slow that I will have to run in a separate machine and etcetera. So separating the test suite in fast and slow makes a lot of sense to me. Because if you remember back in the 2000s, our big discussions were what is a unit test? If I'm testing two classes together, is this 2 unit tests or not? Right?

And I think in 2023 we know that this doesn't matter at all. As long as it runs fast and it gives you proper feedback, that's good. So I like this new way of seeing things and it brings way less space for this type of discussions, right? Because I feel no one can argue that a slow test is better than a fast test. You can argue that maybe like integration tests more than unit tests, but no one can say a slow test is better than a fast test, right? So I like this new way of

phrasing this. Yeah, let's go to the point you mentioned just now, right? Because they were asked some discussions and debates about people preferring more about integration tests rather than unit tests, some think that doing unit tests because you probably most of the things they feel is less valuable. So what is your view about this? I know it's it's probably a little bit more hot topics to discuss, but what is your view on this? I think I have a pretty clear opinion on this, right?

And I think mocking can be dangerous for sure, because if you mock too much, then at the end you're not testing anything. So you want to test as much as possible real behavior or behavior that will look like in production. But at the same time, you don't want to be slowed down by things

that you don't control so much. For example, let's say you're writing a piece of code that makes a call to web service that is developed by another team, and suddenly for you to really run this test, you need that web service available to you, right? And this becomes very quickly a pain. So in this case it's mocking makes a lot of sense and the web service you don't control so much. Let's give an example of something you control a database. Should I mock my database or

not? To be honest, I believe my opinion is that you should not mock your database, because today you can make tests with database so fast that you can actually run them during your pre merge time and you get feedback very fast. So why would you mock something that is not really preventing you from writing the test in an easy way and to run it fast. So I think that's the trade off

that needs to happen. I think it's not about writing integration tests or unit tests, but it's about writing lots of tests that are fast in the end and that you can have full control over. And those are the things you should mock, right? Things that you don't have control over. And then of course, if you go back to the testing pyramid, let's say you're mocking this

web service. So you can write lots of unit tests, you know, super fast tests that just mock this web service, but you can still have one or two or three integration tests that are a bit more expensive that make real calls to this web service. So you see that things work when you put all the components together. But you know something that I always say is you should not write an integration test that could have been a unit test,

right? You don't need an integration test to exercise an if statement in your code. Just write a unit test for it. Leave integration test for what it really pays off, that is to find integration bugs. I really love your pragmatic approach to answering this question so I fully. Agree as well in terms of control, right. So first think about do you control that and especially if not just the. External service or infrastructure. Related time is also probably one thing that you want to

consider. Right time is something you cannot control by itself, but you can actually create like a fake system or mock it sometimes. So I really love the way you explain about this. So let's move on to maybe some of the other techniques right where you have talked about unit tests, integration tests, end to end tests. In your book you actually mentioned couple of testing techniques. So let's start with the first one which is called the

specification based testing. And it's actually very interesting the way you start by expanding this. I would assume like some people would start with unit test, but you start with specification based testing. So maybe tell us more what is specification based testing, Why you prioritize that as the first? Cool, Yeah, So what is testing right? So testing is about trying to find bugs.

And how do you do this? Because you compare what you expect the program to do with what the program does, right? And for you to do this you need to know what the program should do. Where is this information in the requirements? And I'm not saying like a Word document that contains the requirements or UML, it doesn't matter. The requirements can be in your

mind at some point. There's this notion of what the program should do and specification based techniques are the ones that help you look at the requirements and identify interesting test cases. This is not new. Again, this dates from the 80s and we became very good at coming up with techniques. And basically what I showed in my book is to look at the requirements. It's quite easy to see those are the inputs of the program. This is sort of what the program needs to do in the output.

Those are the different paths that the program may take, so on and so forth. And you can look to all of this and then get inspiration to write your tests. And the one technique I show in my book is actually a very basic one that basically says look at the inputs of your program, or the method you're testing, or the component, whatever. What are the inputs? Separate them one by one. Let's say your function receives 3 inputs, an integer, a string that represents whatever, and the list.

Look at them separately and explore their domain. So let's say this is string. What are the possible strings that can come here? Are there any special strings that would make the program to do something different? You do this per input. And why you do this separately? Because it's just much easier for our brains to process small things, right? So you do each one of them separately, Then you try to look at all of them together.

You look for other possible corner cases and so that might be explicit on the documentation or the requirements. And then and only then you come up with test cases. And that is sort of the idea of specification based testing, that you start your tests from what the program should do and

not from the implementation. And I actually like this very much because as a developer, right, the person writing the codes that I'm about to test, that gives me the opportunity to disconnect from my implementation and really focus, hey, this is the input that I'm going to pass through the program. This is the expected output.

So this is my suggestion. If you really want to be a little bit more systematic when it comes to testing is first step is start writing tests or creating test cases from the specs from the requirements. So when we talk about specification based testing, I think people always, I mean not always often associate with BDD right Behaviour driven design and sometimes it's like Gherkin type of language and things like that. What's your view of this?

Do you always have to come up with BDD type of specification test or is there some other types of tests? That we can also use. Yeah, very good question and maybe that's the difference between my book and other books. Usually when people talk about specification based testing, they are focusing on a very coarse grained feature from the point of view of user what should I test?

In my book I show that you can do this at method level for example, or at class level, because as a developer that's sort of the units you're always handling with, right? And I think to me BDD makes more sense when you're looking at the big picture and looking at the whole functionality from really the point of view of the final user and the specification based techniques you can apply in both. Now should you write tests in a BDD style? I think that's really a matter of taste.

I am personally not a BDD fan, so I don't like tests using BDD style. I don't use Cucumber, that's just not my thing. But I see where this comes from, right? And I don't think you know in practice this is too different. It's really telling you to focus on the specs, on the behavior of the program, right? And coming up with the test cases. Tooling is a matter of personal taste in the end, right?

As long as you're looking at the behavior of the program, what you expect from it, I think this is a great step towards a good testing. I hope you enjoyed this short clip from Techly Juno favorite playlist. If you find this episode useful, please help share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you want to listen more from this conversation, please go back and listen to the original full conversation with my guests.

Stay tuned for the next Techly Journal episode, and until then, goodbye.