So we constantly have to keep asking ourselves regularly, like what does quality mean to our end users, to the people that matter, and what does it matter like over time? The quality is connected to the risks and the risks is connected to the testing. If we don't keep on our own quality, our testing and our development will drift because we are no longer building the thing that the
people that matter care about anymore. The Show Hey everyone, my name is Henry Suriawirawan, and you're listening to the TechLegional Podcast, the show where I'll be bringing you the greatest technical leaders, practitioners, and thought leaders in the industry to discuss about their journey, ideas, and practices that we all can learn and apply to build a highly performing technical team and to make an impact
in your personal work. So let's dive into our journal. Hello guys, welcome back to another new episode of TechLegional Podcast. Today I have Mark Winteringham here. So we're going to cover a lot about testing web APIs and also testing in general. And later on we'll sneak in the AI topics part as well because I think it's kind of like the recent trending topics, AI and testing. So Mark, thank you for coming
and looking forward for this conversation. Thank you, thank you for having me on. Right, Mark, I always love to start my conversation by asking you to share your career highlights first or any turning points that you think we all can learn from you. Yeah, sure. So I always start with the terrible joke of I wasn't supposed to be a software tester, I was supposed to be a rock star, but things kind of went a bit
different trajectory. So yeah, I actually studied music at university, music and technology. I was always interested in computers, even as a kid, but I was also very interested in music as well. But my first job was actually
testing music notation. So I managed to carve myself a little bit of a niche of having the ability to work with technology but also being able to read music to an advanced degree that meant that I could sort of get into the testing role. And it was more of a I wanted to get myself a foot in the door into tech because I wanted to write music
for video games. And I'd heard the way through into the games industry was through testing, I couldn't get a testing job and sort of that sort of classic thing of we're looking for a junior, must have three years experience, that sort of thing. But yeah, so I got this job as a tester and actually
found that I really enjoyed the process of testing. And then I was quite lucky early on that I had a mentor who saw my interest in technology and coached me into getting into test automation quite early on. So like most of my career has been around the test automation space, working across a lot of different companies or startups, big
enterprise places. I moved into contracting about five, six years into my career and that gave me an opportunity to move around. And it was sort of about six years into that that I
started teaching as well, start teaching testing. So talking about testing web APIs, my first workshop that I built was around teaching testers how to use APIs, how to test them, how to apply sort of exploratory testing and heuristics and stuff, but
as well as automation. Found that I got really real good taste of that and that rock style mentality of touring the world and going to conferences and teaching and talking about, not just web APIs, but also about testing and
automation in general, as well. I then teamed up with a chap called Richard Bradshaw who's a friendly tester and we started running a course called Automation Testing. So whilst I was doing testing, I was also teaching and learning a lot through various conferences and things I have had. And yeah, sort of about sort of 10 years in, that's when I started writing my
first book, Testing Web APIs. And then yeah, sort of bring up to now on quality engineer for John Lewis partnership. I'm sort of still working in the test automation space, but I'm sort of thinking about quality in general these days, not just about the quality of the product, but the quality of the work, the quality of the testing and supporting my team so that they can be the best that they can be. Thanks
for sharing your story. I think it's pretty interesting testing music notation. I haven't heard it before. So I think that's how you got into this whole testing journey. I think that's pretty interesting to hear. Thanks for sharing that. Hey, thank you for being part of the TechLyJunior community. This show wouldn't be the same without your ears. And
you are the reason this show exists. If you're loving TLJ and want to see it keep on growing, consider becoming a patron at TechLyJunior .dev .patreon or buying me a coffee at TechLyJunior .dev .coffee. Every little bit helps fill the research, editing and sleepless nights that go into making this show the best it can be. Thanks for being the best listeners any podcast could ask for. And now let's get back to our
episode. So you mentioned you have been doing a lot of automation, API automation, web API automation, maybe a little bit of background. How did you come up with the idea to write this book? What kind of problems that you're trying to solve? Well, as I said, it started with the workshop initially. I was keen to just sort of put something out there and
do a bit of public speaking. So I kind of set myself the challenge of building this workshop. And then I went out with a guy I used to run London Tester Gathering. His name is Tony Bruce. And he used to run these collection of workshops. And we were having a drink and I was like, I've got this idea of an API testing workshop. And he was like, right, you're in. You're doing it. And
I was like, oh, OK. So it kind of just sort of kicked off like that. But something that's always kind of threaded in a lot of my teaching is that I like to teach practical skills, but also have a foundation of mindset and philosophy towards how we're doing our testing. So I'm a huge advocate for
exploratory testing. So I love doing automation, but I think exploratory testing is a massively important skill for all testers or for anyone who's involved in kind of quality space. But teaching that is quite tricky. Whereas if I can teach someone to test a web API in an exploratory testing context, then they kind of implicitly learn that sort of information. So that was kind of what I was doing with the book,
with testing web APIs. I developed this body of work, this understanding of how I approach testing web APIs. And at a couple of the temps at trying to write it, people kept sort of encouraging me, but I was sort of like, there's not enough material here. Turns out I was wrong. But I ended up watching some online gaming streamer who was also an author. And someone asked him, like,
how do you write a book? And he was like, one page a day, or a page at a time sort of thing. So I was like, oh, OK, I'll give that a try. So I literally wrote a page a day for a month. And before I knew it, I had my first chapter. I thought, actually, I think I could do this. I'd been blogging a lot previously and writing articles and stuff for companies and teaching materials. So
I'd sort of built up that skill set. So it was kind of, yeah, it was sort of this sort of the training that I was doing on the side, the growing interest in writing. And then, like, yeah, I say, this sort of small gradual process of building something up over time to end up with a body of work. Right. And when I read your book, actually, it's pretty
interesting. I was expecting actually to read a lot of materials about automation, how you deal with, I don't know, postman, rest APIs and things like that. But actually, the way you explain stuff in your book, it's a mix of mindset, philosophy, and also different strategies, right? Which I think in your book, you mentioned it, like the holistic approach. So tell us why it is very important for testing
strategy or testing approach or methodology. Has to be more holistic. And what do you mean by holistic here? So when we take a step back, when we're talking about holistic, we're talking about basically different activities in the context of the testing strategy, different activities that are being executed to address different types of
risks. And that's why I think holistic strategies are necessary, because we have to handle lots of different types of risks that can impact our product. So for example, things like automation is useful, but it is very much targeted at a specific set of risks, which tend to be the sort of kind of the functional, the correctness
of a product. They are change detectors. So they are there to help us determine whether or not the system has changed intentionally or unintentionally. But that doesn't help you with things like performance. That doesn't help you with issues around implementing the wrong thing in the first place, or how those
fringe or edge cases occur in our APIs. All those different types of risks, security, how the end users can use it, how it interacts with other things. All these are different types of distinct risks that we may or may not care about. So quality comes into this. Like what does quality mean to our end users and what risks could impact that? That determines what type of
activities we do. The challenge is that if we go in with what's the normal mindset towards testing, which is running test scripts, having them executed manually or automated, is you get this sort of idea of a monoculture. We're only focused on a very specific type of risk, and we're ignoring these other aspects to our potential detriment or to a negative impact. So there's that aspect. The other side as well is
that, say, every context is different. So if I'm testing APIs for... So I worked for HMRC for a while, and I was on their tax platform. Performance is important. Accuracy of calculations is important. Whereas perhaps a classic one that people were talking about
back in the day was like Pokemon Go. People probably care much more about performance, and they may care more about security, like personal data being shared across, or people stealing all of your inventory from your account or something like that. So different contexts have different needs, different
aspects of quality. So again, holistic being holistic means you're being responsive to what's going on around you, what is that you're dealing with, you're tuning into that, rather than saying one size fits all, trying to square peg it around whole. That sort of idea. Yeah. So thanks for mentioning this understanding about context and risk. So I think that's the main theme all over your book.
So you cover why specifically we need to care about certain stuff, and what kind of risks are we trying to solve here. And you also emphasize that we as testers or developers, we have to understand the problem first before we actually approach testing overall, rather than starting to just write test scripts and things like that. So maybe can you explain how can we actually start by understanding the problem
better, and why is it important? So I'm always a big advocate of if you're starting a new project, is just generally asking questions and exploring, but not necessarily exploring in a way to make judgments. You'll do that at a later point, but it's more of a sort of exploring the product, exploring the people who
work on the product. That's why I quite like the 10 Ps of Testability by Robmini and Ashwinta, because it breaks down a context into these distinct areas. So looking at the people, looking at the process, how is the product built? What technologies are we using? How do we get it from our computers in front of other people? It's like the pipeline process.
So understanding all of that information helps us to, again, better appreciate what challenges we face as a team. It helps us understand who our own users are, and what are we trying to achieve for them. And it's by putting all of this sort of information together that we start to identify those opportunities.
So for me, testing is always about supporting a team, making sure that they're informed and they're making right decisions, they're making the most valuable decisions at the best of times. So yeah, having that sort of context in place, it becomes easier to identify those opportunities to sort of elevate the team so that they can build a higher quality product. So
that's a big thing as well. And I was like, testing doesn't necessarily, like I could do all the testing in the world, but we could still end up with a very low quality product. It may work, but people may hate it, or it may work in a certain way, but as soon as somebody presses the shiny red button on page three, the whole thing
falls over. So yeah, so gathering all of that kind of information helps us identify those opportunities. And then from there, we can start being strategic about which opportunities are we going to follow, how are we going to measure those opportunities and those ways in which we address those opportunities are valuable and we're not going off
track. Yeah, but it all comes from that sort of kind of information gathering process at the start. Yeah, so I think another thing you mentioned in the book after you asked questions and things like that, right? It's very important to build shared understanding among different team members, maybe in the teams, right?
Because I think one common practice I see about testing, a lot of teams actually kind of like throw it over the wall, you know, like they have a team of testers, you know, you pass them the requirements, pass them the binaries, right, and let them test. Maybe it's more like a black box approach, but at the same time, probably the shared
understanding is not there, right? And I think one thing that I really like in the book is about this Venn diagram, and you kind of like bring that all over different chapters, right, when you cover different strategies, maybe elaborate a little bit more about this Venn diagram, how can we
actually use it in our testing strategy? Yeah, so it's based off a model, a visual model, that James Lindsay created to kind of explain the value of exploratory testing, but I remember when he showed it to me and I really liked it, but I actually thought that it would be applied to testing as a whole. So the idea is, like you say, it's a Venn diagram. We have one circle, which is the imagination, and another
circle, which is implementation. So on the imagination side, this is where we are testing to learn about what it is that we want in our product, what do we want to build, and inside that, we will have explicit information, so like say like requirements, acceptance criteria, test cases, documentation, but then we also have implicit and tacit information there as well. So
why are we building this product? When someone says relevant results, what do you mean by relevant? Relevant to who? That's where the sort of misunderstandings come from. So we want to ask questions there to dispel those incorrect assumptions, misunderstandings across the team. And the idea is, the more we learn about that side, we
apply the same thing on the implementation side. So the implementation side is the product, the thing that exists. Again, if we are only testing based on explicit information, so again, acceptance criteria, test scripts, requirements, that sort of stuff, we're only testing a small portion of
actually how the product behaves. So things like exploratory testing and monitoring can be really useful because those sort of activities help us learn more about how the product actually really behaves. So by learning more about how the product actually behaves, and by learning more about how we want the product to work in the first place, the more we can overlap these two areas
so that we can make better informed decisions. Like I said, if we go for just a monoculture approach, we're just running test scripts, then you get a little bit of an overlap in that FEND diagram because you are kind of using your explicit understanding of what using the product is used to test the product, and you learn some information, but
you are missing out on so much more. So that's why I like using that model, because it communicates for me the goal of testing, which is to find out as much about both of these items and get that overlap as much as possible. It'll never be 100%, but you're always striving towards it.
But then also as well, again, it goes back to this risk aspect of some risks live more in the implementation side, some live more in the imagination side, and then some exist in that sort of overlap because as our product gets more complex, things like regression become an important factor, but they're not the be all and end all. Again, it's that spread of all of them. So yeah, that's why I'm sort
of a big fan of that model. And as you say, it's a great teaching aid in the book because I can move across the model in different places and talk about how these different activities
work in different areas. Yeah, and I think not to mention also for different areas, like the imagination will have some testing strategies that kind of like more focus towards that, things like for example, contract testing, right, and also test API designs, and also on the implementation side,
maybe there are other aspects as well. So you kind of like bring the whole holistic approach to testing, and it's not just like an automation in terms of API step by step, right? So I think it's very interesting definitely. And in the overlap, right, so if we can make it larger and larger, we kind of like align both the
imagination and implementation. And I think that's where the testing strategy, where you build a lot more automation to cover both areas, I think will be more powerful. So I think throughout the book, you will see a lot of these Venn diagrams. So for people who are interested, you can check out the book as well. I think it's really powerful framework, frame along all
these testing strategy. So after we understand about the importance of this testing approach and understanding context and risk, right, we need to pick the testing strategies. And I think you emphasize a lot in this conversation
already as well about the risk. I think sometimes this is not intuitive for many teams, I think, because when we talk about testing, right, they always come up with, okay, what are the requirements, right, what users need to do, but they actually don't talk it in the perspective of risk. So maybe if you can explain a little bit more, how can we actually start building our test strategy using
this risk perspective? Yeah, I think it's as much, like you say, it's about mindset as it is in terms of specific techniques and approaches. We're always under so much pressure to deliver. And I think because of that, we tend to sort of focus on the output at the end, the artifact. So when we talk about the context of testing, we're
talking about the tests that were done. So I think it's like when we're trying to get people to do sort of risk based testing, the first thing is really to get this, it's funny, I've been seeing this quite a lot lately again, and it does annoy me as it's talking about testing types, types of testing. So people build strategies or
approaches to testing around testing types. So you have to do integration testing, you have to do functional testing. It's like, yes, those are types of testing, but that's, you know, it's trying to think of a good analogy. It's like talking about types of tools, like, oh, you know, we're going to do some DIY. Well, I have to have a screwdriver. I have to have a hammer. So
it's fine, but you're painting a wall. So those things aren't going to be very useful for you. So I think, yeah, it's that mindset of trying not to think of it as types or boxes of testing. Think about the product, think about the end goal and what risks impact that. Then in terms of like the next sort of tricky part, he's getting people to be sort
of kind of risk based and risk analysis. So this is where like a tester or anyone who's interested in qualities, like number one tool is questions. He's asking questions and going, you know, what will happen if X, Y and Z happens? Or what would be the result of this? What do you mean by that? So using like five Ws and an H, Y, Y, who, when and how. Yet those sort of primers to ask questions is a great
place to start. Then you've got frameworks like risk storming by Barry Vandal, who has built this like brilliant online asynchronous tool that you can use. He's literally risk storming online. Just Google that and I can't URL. Sorry, Barry. But using sort of kind of more sort of heuristic based techniques. So Elizabeth Hendrickson, I think, came up with the idea of the newspaper game.
So imagine a newspaper article, like a headline in the paper and use that as a trigger to what would cause company X leaks customer data, right? Well, what happened to cause that headline? And you sort of follow the story there. There are frameworks that we can
use that we can follow. So like risk storming is really good because it helps you identify quality, then it helps you identify risks, and then it helps you identify what's testing you want to do for those risks. Whereas, you know, that's much more structured workshop based thing. Whereas, yeah, like the headline games, things like test a
bleep on my talks as well. Those are sort of much more informal things like five Ws and H. It's just a primary trigger to keep to ask those questions. Yeah. So I think if people adopt this kind of a risk based approach, I think it will be much more valuable in terms of the output of your testing activities, right? Not just producing, you know, hundreds of test scripts, just UI automation, for
example, or just user based approach. But you kind of like miss the whole aspects of different risks, right? Because we cannot test everything. Definitely, there will be too many. But yeah, how should we prioritize what kind of things that we should test? I think this risk based approach is really important. And before we actually come up with those prioritized risks, right, you also mentioned that we have to
know about the prioritized quality attributes. But let's say, but we talk about quality first, right? Because I think the whole purpose of testing normally is to a certain standard or quality of the product. So what do you mean by quality? And how do we actually think about the quality attributes? Yeah, so I think that's the crux of it. Of all of it is that, yeah, like what does quality
mean? So whenever I'm sort of like people ask me about, oh, you know, what is testing about? And why do you care? I always go with that. That's a good question. So of, you know, what flavor of crisps or chips do you like? And they go, oh, I like ready salted or pre -cured or something like that. You're wrong. It's salt and vinegar because they're my favorite. And I know quality and you
don't have quality. And they're like, no, no, you're wrong. Like this flavor is that, you know, we are all individuals and we're all contextual in our own rights. And the same thing can kind of be applied against the products. So when we think about quality, we have to think of it not necessarily as a singular thing, but as like this multi -dimensional thing. So there are different types
of characteristics. So I mentioned some earlier when I was talking about accuracy and results and responsiveness, there is a great list of quality characteristics from test eye, which you can Google and it must have something like 50, 60, maybe 70 plus quality characteristics out there and stuff. And some of them are, you know, technical based. So does it, is it operable across different devices? Does it integrate with
environments and stuff? But then some of them are much more kind of like a motif. So does it feel good to use? Does it look good? Does it make me excited? So lots of different characteristics. Then you have kind of the time factor of this is that different things will matter at different times. Another factor to take a step back is different quality
characteristics will matter to different people. So our end users might care that it looks good and is easy to use. But if we're in a regulated environment, our auditor wants to make sure that it's got quality characteristics of we can understand how it's working. It's got good auditing processes, that sort of thing. So different people have different perspectives in that way as well. And those things change over
time. So if you're a startup, what quality means to your end users and to your stakeholders is going to mean something very different when you're moving into like a small medium company to enterprise. So we constantly have to keep asking ourselves regularly, like what does quality mean to our end users to the people that matter? And
what does it matter like over time? Otherwise, again, the quality is connected to the risks and the risks is connected to the testing. If we don't keep an eye on quality, we again are testing and our development will drift because we are no longer building the thing that the people that matter care about anymore. So yeah, that's kind of why we want to think about
quality characteristics on a regular basis. Yeah, some people actually point the terms quality attributes here like non -functional requirements or performance attributes or there are multiple terms about it. But essentially, you're looking into like the elities. So some people also call it elities, availability, scalability and things like
that. So I think this quality attributes sometimes in some teams, they tend to under prioritize that and only focus on the functional aspect. Like given certain input, what's the output that should come out? And they just create more test cases around that. But actually, when you have holistic approach, so you will have aspects of quality attributes that you
want to test as well. And again, depending on the product, depending on the kind of quality that the team or the user cares about, you build some kind of testing strategies around those things as well. So I think it's really important for those of you who would love to ascertain your quality of the product, first of all, know about your quality attributes, quality characteristics, and then
build some kind of testing strategies around those. So let's maybe go into several testing strategies that you cover in the book. Maybe some from the implementation, some from the imagination. And later on, we'll also talk about automation testing, definitely. So the first strategy that I want to pick is actually testing API design. Personally, I find this a bit rare being covered in the development space or at
least in my area. So what do you mean by testing API design? How do we do that? And why is it important? So I think it's interesting that you say that you don't see that very often, because I think it happens all the time, but it's not done in a explicit, structured, or clear format. So I've worked with developers and they're doing this stuff implicitly. They're asking those questions as well. And
that is a form of testing. So when we're talking about like testing API designs, this is where this whole sort of shift left mindset comes in. Again, it's that testing the ideas, testing the assumptions. So we're in a situation, perhaps, or doing something like a collaborative session with different people. So that's all kind of three Amigo style things or just again, something informal, but we're together as a
team and we're looking to implement something. So what we're thinking here is what is it that we want to build? What is the solution that we're proposing? And then it's asking questions about aspects of it. So you're asking questions around maybe the actual sort of technical implementation. Maybe you're asking questions around, oh, you've set these business rules. Do you mean like, is the boundary here or is it
there? What do you mean by this sort of domain phrase? I love it when someone says, I just want to ask a stupid question because they're the best questions because they're usually because someone's like, I don't understand an aspect of this. So I need clarification. There's nothing more satisfying when you ask that three people answer the different answers
because everyone's got that sort of misunderstanding. So it's about understanding the implementation, raising risks, raising test ideas and stuff. So, oh, I see that you want to put a limit on this field. What happens if I go over that limit and they go, well, you know, you should return a 500 status code. Shouldn't it be like a 400 because it's a user input error and stuff and it's conversations leads to that.
So, yeah, like testing API designs is very much about just making that process that everybody does, making it more explicit, making it more collaborative. So getting people to share ideas so that we come out, we have a clear idea of what it is that we want to build and also that we are all on the same page. Like we all agree. I remember seeing some
research once. I wish I could find it, but there was research that showed that most bugs come not from errors in code, but from misunderstandings of requirements or of user's needs. So I think that's why this sort of aspect is essential. And it's useful for a testing perspective as well because you can ask the questions then of, how are we going to test this and what things do we need to do to make this thing
testable. And then, yeah, you can factor in different types of tools. So like things like swagger documentation is really useful because it's a way of documenting the API. You know, when it makes it readable, you can trigger off ideas and questions from and conversations from,
but it's not implementing the whole code base. And then it's got its own sort of ecosystem of tools to make it easier for people to translate that into code at a later date. So you get a kind of double win there of everybody's on the same page. We've got this explicit documentation, press of a button, attention to turn this, get us halfway down the line in terms of implementing a new endpoint
or a new API. Right. So I think this approach, when you mentioned about collaborative activities, right, I think again, it's rare to me always, you know, seeing the three Amigos in practice, right, like maybe the business analyst or the product owner, right, with the tester and developer, always collaborating actively for almost all of the stories or, you know, the requirements that they're
implementing. So I think the key again here just to explicitly mention it, collaborative software development, I think is really, really crucial, right, because a shared understanding, you don't want to misunderstand the software requirements or expected behavior of the product that you're building, right. So do a lot more of these
testing API design, right. Maybe it's not just API design, sometimes it's driven by the UI, right, for people who are building UI -centric journey. And I saw talk about it from the API design perspective, but this is a mindset and approach that we've applied to our team in general. And like, it's something that's been championed by Lisa Crispin and Janet Gregory, their agile testing books, all of their work on
continuous testing as well. And they're big advocates for holistic testing as well. But they've really sort of kind of opened up that mindset of getting teams to talk to each other about the things that they're building, the quality that they want to imbue into the things that they build. So yeah, you can apply it to any sort of kind of context that you're working in. Yeah.
And not to mention also, after you get that shared understanding, please write it down somewhere, maybe as a documentation or maybe some kind of a test specifications, whatever that is, right, so that the shared understanding is not lost when people change or maybe business strategy change and things like that. So thanks for sharing about
this testing API design. So the second testing strategy that I'd like you to maybe talk a little bit more, which is what you mentioned in the beginning, something that you have passion about exploratory testing. So what is exploratory testing and how can we approach this kind of testing? So exploratory testing for me
is a semi -structured approach to testing. It basically relies on my creativity as an individual, but still has enough structure and enough boundary and place that we are focused on what we want to do. So I very much enjoy sort of charter -based approach to exploratory testing, which I've mentioned a lot in Exploratory by Elizabeth Hendrickson. And the idea is, again, so
it's connected to risk. If I identify the risk, I set out an exploratory testing charter to learn about that risk, and then I will do testing. And it's not scripted. It is very much me following my instincts, using my own internal heuristics, maybe using some external heuristics as well, taking notes, reviewing those and looking at sort of paths
and avenues towards how I will do my testing. So it's not great for repeatability, but it's really good for sort of expansive broad testing that's going to reach into areas that scripted testing is not really going to work on. One of the main reasons why I love exploratory testing as well is if you're a good exploratory tester, it starts to blur the edges of what automation means.
Some of the most successful things I've had in the automation space have been
within the context of exploratory testing. So yeah, it's great building automated test cases and test scripts and stuff, but if you can build a tool that can scrape your monitoring system or your log files to let you know when errors occur, if you can build a little script that injects all of the fields, like if you've got a massive like 20 input form field to fill in, but you only need to test one
field or something like that, building a tool that can inject data into that, tools that can set up data for you really quickly. I find that quite satisfying as an activity, and it allows me to accelerate my exploratory testing because I'm less focused on the setup, than more into the sort of observation and analysis and reaction to those sort of things as
well. I think the charter is very important when you do this exploratory testing. Exploratory doesn't mean that it's super, you know, like freedom, right? You can just do whatever you like and figure it out if you find any bugs or not. But I think the most probably more effective way is
actually to set a charter, right? Again, it's driven by the risk that you identify earlier regarding your quality attributes and things like that. And then you kind of like set the charter, you explore based on that theme and
see how it goes, right? And some people and some teams actually do like bug bashing, you know, like group activities where, you know, you set maybe a theme, then they will just explore the application, try to break it and report
issues that they find along the way. So I think for people who would love to do exploratory testing, so one tip here, very, very important is to set it based on the quality characteristics and also set a charter before you start it. Yeah. And so I wondered, I had one last thing to that as well is that you can go
off charter as well. So if you find something like some astonishingly bad part of the system elsewhere, there's nothing wrong with going off charter as well. The important fact is that you know that you've done that. And then you can be like, right, well, I've discovered all this other interesting stuff. Now I'm going to go back to the thing that I wanted to
focus on. Whereas I will go look at the other thing and then go, my work here is done and forget about the initial thing that we were working. So again, like risk is your guide, but it doesn't set strict boundaries on what you can and can't do. You do have flexibility and fluency in it as well. Right. Thanks for adding that. So I think you can go off chart. But yeah, don't forget to,
again, document your findings, right? Don't just take it like ad hoc activity and that's it, right? So speaking about testing, you know, obviously people associate that a lot with the automated API testing, right? Sometimes it's called end to end testing or acceptance
testing. Maybe tell us more how can we do better automated testing because I'm sure many people understand about automated testing, but how to do it more effectively or better is something that maybe you can give some advice on. Yeah.
So it's interesting, like sort of drawing the parallels between end to end testing and automation testing because I think that automation is literally just using tools to do something that we were doing. So one of the big things, again, with automation is focusing on risks. So I have these acronyms or these sort of phrases that I use regularly to sort of help me understand what I'm automating. I have this tutu and Tata.
So from the user interface, am I testing the UI or am I testing through the UI? A lot of people that work in the automation space, especially testers, will do things full stack and they've got some automated tests running, but the automated test is actually focused on a risk in the back end, like some business calculation. You
don't need the user interface for that. You could do on the API layer, which is why we have Tata. Am I testing the API or am I testing through the API? Again, if this is some sort of business calculation, some sort of rule, then couldn't I build a unit test for this? So I think sometimes like the success with API automation or success with any automation is almost the inverse of saying, I'm not
going to automate it on this layer. I'm not going to automate that thing. I'm going to look at talking about end to end tests. I'll look at all the different parts of the system that I'm touching with my end to end test and I'll try and be more targeted. However, then what I mean is the risks that we do care about on the API are much more focused around like how it's presented, how
it's structured. Does it respond in the correct way that we want it to respond? Contract testing sits under the banner of automated testing because it's using tools again. That's very much focused on contract drift between integrations as well. Performance testing is technically automated testing as well.
The oracle is slightly different in terms of how we determine what's good or bad, but we still need to use tools to put an API or APIs under load as well. So, you know, Stein -Salica stuck record, but again,
it's all about risk. Like going back to what I was saying earlier with being more specific and targeted, if I have tested every individual component within my API with unit tests, then my API tests are much more about do these things integrate, might receiving the right requests and sending the right responses. So, what you end up with is probably less API tests. Again,
context matters here. I've worked on projects where I can't go to unit tests because everything already comes to us compiled. And I don't just mean me as a tester. I mean, literally, the whole company gets given an application that's already built and then we're adding onto it. So, we have to do everything on the API layer, but we're making that informed decision based on what are
the risks, the context as well. So, I think, yeah, a big thing with automation is that you're not trying to be exhaustive on the API layer. You're trying to be selective in terms of, again, the types of risks that we care about and assume that other testing activities are going on, maybe on the more atomic level. So, bringing it back to end to end testing, what my goal for an end to end test is, does everything work
end to end? That's literally all my test does. Does this bit, the front end, talk to the back end? Does API talk to API B, C and D? But I don't really care about the business logic in them because I know that's been checked or tested in a previous way. I like one heuristic that you mentioned in the book,
right? So, if you have the capability to push your checks or your testing lower, you should actually try to do that more often, right? Because if you're implemented in the upper layer, most likely, it's going to be slower, less maintainable, and also more complicated to set up, right? So, I think this is very important for people not just focus a lot on the end to end testing or API testing, the 2 -2 -tata
thing, I think that's also very important. Are you testing the API itself? Or are you testing through the API, which is kind of like the business logic or the validation and things like that, right? So, I think that's kind of like a good heuristic as well that people can use so that we can have a more holistic approach to
testing. And I think I also like another thing that you mentioned in the book about automation testing, because many teams actually drive the API testing by coverage or just user requirements. So, again, I think you mentioned it several times already, use risk -based approach, don't just focus on the coverage, right? You know, like aiming for higher code coverage or test coverage, right? But also look at the different
type of risks that you cover. I think that's very important. I was going to say, like coverage of risk is a type of coverage. What we're trying to do here is not necessarily cover every instance of data, every code path, or, you know, different types of coverage, like, you know, is some tool reporting factor. You've mentioned it, but how has it decided that number? There's nothing wrong with the word coverage. It's
more about what are you covering? That's why, yeah, I think what we should be covering is the risks that matter to us, not necessarily the amount of paths we have for what some sort of tool is telling us is acceptable. Right. So, maybe speaking a little bit on the requirements aspect, right? Because you mentioned maybe a lot of bugs are introduced by a lot of misunderstanding
about the requirements. So, how can we get less misunderstanding? What about this acceptance driven testing that some people advocate about? Do you have some advice on this area as well? Yeah. So, it's an interesting space, like acceptance test driven design and like behavior driven design as well. It took me a long time to sort of get my head around it, really. You have to kind of break it down into smaller chunks.
So, we talked about like behavior driven design or behavior driven development. We've already kind of talked about one of the core tenants of BDD already, which is shift left testing ideas, questioning designs,
doing that work together collaboratively. So, we all have that understanding from there, as you were mentioning earlier as well, then we want to capture that into some sort of documentation, some sort of kind of concrete examples and scenarios that describe how we expect things to work. And then it's from there that we can use that for acceptance test driven design. I always think that it's really interesting with ATB.
It's because the way that the risks that I say that that's mitigating aren't functional business risks, it's more team risks. So, yeah, I was running a workshop on automation and this tester who was there turned around and sort of thought they'd ask a cheeky question. They said, oh, you know, it's all well and good me automating all this testing, but can I automate my developers so they actually build the right thing? And
I was like, well, it's fine. You should say that because that is the risk that I think like ATB is mitigating. If you follow that sort of red green refactor approach with acceptance test driven design, what it's doing is it's putting the boundaries in the place to deliver the right thing. So, it's giving some sort of cue to the developer of you haven't built what has
been asked of you until you make it pass. But it's done in a way that because it's described from a business level or from how a user will interact with the system, because it's done from that sort of that level, it still gives the developer scope to implement it in the way that they want to implement it. So, it's not so prescribed that all kind of creativity in the development space is
gone. I like to think of it as like putting the barriers on when you're bowling, you know, it stops you from bowling a gutter ball, but you can still hit one pit or ten as a developer. So, again, it's this holistic thing going on here is that ATDD is really useful for those risks of making sure that we deliver the right things. That doesn't mean that it's going to cover all of the
other risks by data risks and structural risks. And that's why we might have other automation or other testing activities in place. So, yeah, I'm a big fan of ATDD, but sometimes it gets kind of a bit lost in the testing soup as a whole. And, you know, you start seeing people start using these frameworks to try and exhaustively test everything or exhaustively set all
these tests out there, developers to follow. And I think that's where you get fatigue and people stop using these things because you just end up with too many tests. You're not focused on the risks of delivery and just end up sort of overwhelming the team and they lose interest in stop doing that approach. Yeah, ATDD definitely has a lot of things to cover. So, you mentioned some important
aspects like it has to be collaborative. That's the first thing, right? Use examples to specify the behaviors, right? Also, like the living document aspect, right? So, I think it's also powerful in that aspect. You kind of like write down your requirements, but also evolve it along the way, right? It's not just a test script, but it's much more beyond that. So, I think for people who love ATDD, do check it
out. I also covered previous episodes with John Ferguson and Jan Molak. So, speaking about the thing that one of your audience asked you just now, right, Chikili, right? How to automate developers. I know that you are writing a new book in the process, AI Assisted Testing. Maybe we can use AI a little bit. So, tell us a little bit more snippet, like why are you writing this book and what should we expect from AI
to help us in testing aspect? Yeah, so AI Assisted Testing, it's almost complete. It's kind of a bit of a logical next step from some of the things that I've been talking about. So, you know, I was talking about, like, exploratory testing and how I use tools within my laboratory testing to aid my testing. The general crux of the new book is around how we can use AI ultimately to assist us to support our testing.
These things like, you know, large language models, Gen AI, they're fantastic. They do these amazing things and they've sort of kicked off this whole conversation not just for testers, but for developers as well of is AI going to replace us. But I don't think that's the effective way of using them. I think the effective way is to reflect on what you do in your role and look at the aspects of it that can
benefit from the use of these types of tools. So, for example, things like generating large sets of data using large language models for those. In the automation space, can I do things like page objects and boilerplate code? Can I feed it some HTML and then just get my page objects out of it straight
away? It's exploring the ideas of, like, you know, how could these tools potentially be trained on our contexts or tuned towards our context as well, so that not only when we ask them a question, do we get a response, but we get a response that is kind of informed by what's going on around it? Because a lot of the tools that exist at the moment general generative AI, we want
something that's been more specific. So, yeah, the book's very much about like how we use these tools. It's exploring things like prompt engineering, fine tuning, our retrieval argumented generation. But it's also as much about us as individuals. How do we identify the places that they can be useful? How can we have like healthy level of skepticism, but not be so skeptical that we become cynical about
these tools as well? And yeah, generally just sort of work out ways in which we can use the general AI tools to help us enhance our testing, you know, test faster, test deeper, that sort of idea. But certainly not replace us, that's for sure. At least not now. Right, I think a lot of people are thinking, you know, scarily that these kind of tools will kind of like effectively eliminate their
jobs or their roles, right? But I can also see some kind of evolution, right? Maybe developers will be able to write more of these test cases, right? Or maybe testers can also use that to improve the code itself, not just reporting bug and let developers fix those bugs, but also kind
of like help to fix it along the way. So I think what kind of things that maybe after you play around with this AI, what kind of things that probably would change because of the introduction of AI, not talking just about elimination of roles, but I think, I don't know, like an evolution of skill set or maybe a different approach of collaboration
altogether. Yeah, Gen AI stuff has been around fairly recently, but like there's been a lot of sort of AI work, machine learning work that's been done in the past that we, especially in the testing specs, we've seen around things like visual comparison tools, for example. So, you know, I get asked this question, like, you know,
well, AI take our testing jobs. And I always want to say, like, if they take our testing jobs, then we have bigger problems to deal with than our jobs right now, because distinctly testing and quality is a human driven, heuristic driven sort of matter. But I agree with you, I think that it has the impact to evolve our roles, I think probably more on the developer side than on the
testing side, as it stands. I think there's a lot of talk about how actually, as we start to rely on tools like co -pilot as developers to build our frameworks, then there's an argument that it's because, you know, it's garbage in
garbage out. So if this thing is trained on bad patterns, it's going to output bad patterns, which is, you know, means boom time for testers, because we're going to have more bugs to find and more testing to do. If you kind of cynically look
at it from that sort of angle. So there is that sort of challenge of how do you test the product that has been developed by not just an individual or collection of individuals, but also by a highly probabilistic machine as well. So
there's that sort of factor. I think that, yeah, the developers who are having success with these tools, and I think it's going to apply in the sort of testing spaces, are the people who are using it for very specific tasks, as I say. So I'm on a Slack group with some developers who are working with things like co
-pilot. And what they've said, what's really interesting is that they can quickly knock up some code, knock up some tests, and then it gives them more time to sort of play around and refactor the code. So to give it that extra polish. So there's almost like a Pareto's law thing going on, where the tool gets them 80 % of the way, and then that individual sort
of factor comes in there. I think as well, there's going to be rolled like as these tools become more and more prevalent, there's going to be more interesting people who can write prompts in people who can engineer these type of tools as well. So not necessarily data scientists and AI scientists, but can you tune a model? Can you build a rack setup? And conversely, you know, for
testers, can you test these systems? And how do you deal with a deterministic system versus an indeterministic system? We've all been taught how to build test cases. You can't do that in this context. So there's new skills to kind of learn there. I think on the not so great side is, and I think that this applies to quite a lot of industries as well as not just software development is, what does
this mean for people coming into the industry? You know, it's all a good me talking from a perspective of privilege of 20 years of experience working in the quality space. But how does someone is just coming into testing? How does someone is just coming into development, learn those sort of important skills? And where do those jobs come
from? Because, you know, if you've got one large language model that's doing the work of 20 junior developers, that's probably going to mean that they're not going to be hiring 19 of those in the future. So I think there are questions around that sort of aspect of moving into this
industry and how that will impact it. I think, yeah, for a lot, it's going to be this evolution of like, how do you build a relationship with these tools in a way that utilizes both your skills and skills and tools? Because if you're not using them, you're not going to be as
performant as the person who is. And if we rely too much on these types of tools, they're just going to end up producing stuff that's not very valuable to us, because they're not designed, they're not chained to our context. Yeah, you brought up a very interesting point here for many of us to reflect, right? So these LLM tools, the JAN
AI tool is currently still undeterministic, right? So it's highly likely when you ask a question, it will come up with a answer, then you ask again, it will come up with b answers, like variant of it. So I think it's very important as a testers, your job is again to a certain quality. So how can you, as a certain quality, if you are undeterministic, so to speak, right? So I think the job, the role
of testers most likely will not be eliminated. I may be wrong. So maybe one day we can get a much improved AI. But I think here the key is to use AI as an assistant, right? It's not to replace anyone and use that to boost our productivity or kind of like to cover all these boilerplates, like generating data or kind of like starting the code itself. So I think be more optimistic. And
I think we are looking forward for the book. I hope to probably also read that book. And if we can also cover another episode for that. So thanks again, Mark, for this interesting conversation about testing. Unfortunately, we have to wrap up. But before I let you go, I have one last question. I always ask my guests, which
I call the three technical leadership wisdom. So if you can think of it just like an advice that you want to give to us, maybe if you can share your version for us to learn from you. Three things to think about. So I think obviously the first one's going to be context is key given that I've banged on about that for nearly 600 pages across two
books. I think that is a truly important aspect is that there is no one size fits all solution to everything. I think as well, you know, from the perspective of being a tester, from being a quality engineer, I'm still learning this myself, but being sort of a
servant leader is massively important as well. So responding to what's going on around you and helping people with problems is going to be much more productive than kind of applying your own will for your own perspective on things as well. And as well, like I think we're talking about the AI stuff as well is reflecting on yourself, reflecting on
your abilities. And so I'm a big advocate of continuous learning and rules to read new stuff, expose myself to new things as well, not just stuff in the software development space, but in the arts as well, I learned loads from in terms of teaching and
presenting from stand up comedy. I've never gone up on the stage and done a set or anything like that, but there's a lot we can learn from people as well and draw those skillsets in as well. So yeah, that would be my three. Wow, it's very interesting. So first is about context, right? Again, like no one thing fits all the kind of a problem,
right? So I think I also like the last part where you kind of like learn from different various industries or different kind of philosophies or comedy and things like that. And you can infuse that to your role. I think that's really powerful as well. So Mark, if people would love to connect with you, ask you more questions, maybe is there a place where
they can reach you online? So usually hovering around on LinkedIn, so you find me as Mark Winteringham there. I am also still on Twitter slash X, what's left of it. That's a two bit tester with a number two. And also have my website, mwtestconsultancy .co .uk, which I hope to be adding some more stuff to once I finish the second book. Right. I highly recommend your testing web API book. So it's not just testing web
APIs guys. So it's also holistic approach, right? Starting from identifying quality characteristics, race approach and things like that. I think it's kind of like comprehensive. And I'm really looking forward for the second book, AI assisted testing. I think everyone is excited about potential AI. And I think we are looking forward for some tips from
you on how to use it more properly. So good luck with your process of writing the second book. Thank you very much.