[German]
[German]
[German]
[German]
[German]
[German]
[German]
Hello, Paul. Nice to have you here on the show. Thank you, Richard. I'm very pleased to be on it. Yeah, great. Yes, we just met a few weeks ago at the testing retreat. The second time we did it together. And, yes, for me, you are one of the heroes of software testing, one of the big guys that did a lot of conferences, a lot of stuff for the testing community. But maybe not everybody in Germany and Austria and Switzerland will know you. So maybe tell something about you.
What are your main parts of your career of the last 10, 20, 30 years? Well, more than that. Well, I started a long time ago in the early '80s when I first started working. I was an engineer, very briefly, but a civil engineer. And I joined a civil engineer's consulting engineer's office. And the boss came over and said, "We've got no work for you, but here's a computer. Here are the manuals. Go and have a play. Tell us what we can do with it."
In those days when, you know, engineers were hands-on computing programmers as well as engineers. And my fate was sealed. And I've been in IT ever since, in effect, although I did work a little bit of engineering, but not much. Anyway, over the course of 10 years, I was a developer, team lead, did a bit of project management, and then got into testing about 30 years ago. And I've been in that ever since. But I've always been a developer. I still write code.
And we can talk about that a little bit later, I'm sure. But in the testing world, I've done a bit of everything. You know, I've done a lot of conference talks, been a consultant, test strategies, a lot of training, coaching, mentoring, that kind of stuff. I've worked with tools. I've written a few tools, and so on and so forth. I've written a few books as well. And perhaps people know me from the conference circuit where, you know, obviously there's publicity associated.
And the books, which I used to give away, not so many now. I need to update what I've got now a little bit more. But I usually give the little pocketbooks away. I've written one big book with a friend of mine, Neil Thompson, as you know. You know Neil. And five little books. And little books are much better because they're only so big. People actually keep them and read them. But anyway, so I've done a bit of everything. I've tried to think more carefully about what the future is whenever.
So going back to, you know, moving from green screens to GUI screens, client server, the internet when it came along, e-commerce. I didn't -- I tried to be ahead of the game and be like first to market with a class, you know, testing client server, testing GUIs, testing the internet, internet applications, web applications. But with mobile, I completely missed that boat. I thought mobile will never add up to much. I mean, they're just toys, aren't they? So I never -- how wrong was I?
But apart from that, that was kind of a mistake, needless to say. But I do a bit of everything. That's the thing. And that's what I enjoy. And that's where I am in my life as well in that I'm not working so hard to make money and sell things except to get involved in interesting things. And, you know, we'll talk about AI no doubt a little bit later. But AI is what I'm interested in right now. Yeah, yeah. Yeah, that's a part we'll talk about today also.
And I think you have a great impact with all your insights and your talks also to me and the conferences and what I wrote about -- read about you. Yes, and we talked upfront of what can we talk about. And you had a testing retreat you mentioned like a pattern or like an acronym on how we can check and prove requirements. And maybe we can deep dive into that. Tell us something about that, please.
Well, one of the little books I wrote -- and I can't see it on my shelf, so I can't show you -- but with another friend of mine, Susan Windsor, we put together a little book which talked about using stories as a mechanism to improve requirements and to link with test automation, you know, through the BDD process, if you like. But we were quite focused on requirements and trying to get requirements right and using stories to help us improve our requirements.
And by stories, I mean Gherkin-style stories. Now, it could be -- it doesn't have to be Gherkin, you know, the DSL, the name of a specific language, Gherkin. They could be just any structured language would work the same way. But the general idea was to try and take a narrative text to develop -- derive stories from that narrative text and then use those stories to, if you like, provide examples in the specification by example way of communicating what requirements are about.
But also, many years ago, my first conference paper, 30-something years ago now, first jurist off, 33 years ago, I think now, 32 years ago, whatever it was, I talked about testing requirements and using -- you know, looking at how we can improve requirements by actually not just doing an inspection, which is a checklist against things that might be interesting but quite often aren't. They're all very low-level and detailed things.
I felt that there was an opportunity to almost prototype an application by automating a prototype directly from requirements. And I had what I called behaviors, tabulated behaviors, but actually what's much better is Gherkin. And about 12, 13 years ago, I kind of came back to that idea, but the idea was to use Gherkin stories to drive a review mechanism to test requirements.
And I came up with this acronym DEFOSPAM, if that's what you were looking for, which is a terrible name, but the initials, like D is -- very quickly, D is definitions, F is functions, features, O is outcomes, S is scenarios, P is prediction. You can predict a behavior. I could talk about all these with more. A is ambiguity. Look for ambiguities. And M is is there anything missing.
So that was the -- and literally in an hour, I sketched out this kind of idea, and I looked at the first letters of everything, and I thought, oh, I'm going to call it DEFOSPAM. It was just a complete accident. And it never sold. So the little book, I think we put it out in 2012, so 12 years ago. We only ever gave that away because we were trying to sell a methodology, and that was part of a methodology to use -- it was basically, I would say, a more comprehensive way of looking at BDD.
Now, the BDD guys would say, oh, no, no, but they're agile. Well, I'm saying you could use BDD in waterfall projects as well as agile projects because we were looking at requirements as narrative text as well as just looking at stories. And stories, examples are fantastic, but they're not comprehensive. They are example-driven. And you need an infinite number of tests to assess software. You need an infinite number of examples to specify software.
So specification by example is a partial solution in my book. I think you need more than that. Yeah, yeah. Let's go deeper into the parts of, you said, definition and function. What do you mean with these words according to the requirements? What are the questions behind it? Okay, so I'll try and get them in order. So D is definitions.
So one of the problems we have with requirements is people will look at requirements documents, read them, and pick some ideas that they don't understand or don't like or do like, but they never question the definitions of words, of terms.
And so what I've seen over the years, over many years, quite often the conversations that are triggered by getting to the bottom of a definition, like what is a customer, what is a product, what is an order, what is a failure, we have the same problem in testing. We've all got personal definitions, which we love, but aren't common across the industry. And so there are really bad arguments about testing by not saying let us agree a set of definitions. Now it sounds very boring and time-consuming.
Well, yeah, it is. But if you do that, at least you know your message is getting through and you are understanding someone else's message. So definitions are critical. And a lot of what AI is about is the large language models, they're all about matching words that have patterns of existence, if you like, in narrative text. And we don't question the definitions of these words. So AI doesn't help us with that.
But what if we use AI to identify things that need definition and maybe to get us started, it could generate the definition as well and say here's a suggested definition for this word that seems to occur an awful lot in your requirements. So D is for definitions. So let's get a glossary of terms together and AI I think can help with that. And we can talk about how AI can do that if you like a bit further on. So the first activity is identifying things that need a definition.
I think that's critical. F is features or functions, don't mind, whichever, which is like what are the features that we're thinking about? How do we package the scope of smaller components of the overall system and call them features, things it does? We need to identify those. So that's kind of okay, so we need to from a requirement or a fragment of a requirement, not a whole document, say, we identify a feature. Now stories are usually defined around features in that respect.
So we'll have a feature, update an order. And you have in Gherkin, you'd have the as a kind of user I want to update an order so that I can escalate the order and get it processed. I don't know what the purpose of it might be. But features are the center of everything. That's the hook. Everything else hangs off, if that makes sense. So that's F, feature. O is outcomes. So in your requirements or in your stories, the system will do things.
So if I input A, B, and C, the system will do D, E, and F, maybe, whatever the requirements are. So outcomes are like what does the system actually do? Can we identify those from the requirements? So that's O. Next one is S. S is scenarios. So what scenarios trigger the outcomes that we can define? So we have a list of outcomes, maybe, but there are also a list of scenarios that our stories, we're talking about stories now, would identify. So situations that the system has to deal with.
And if the system encounters this situation, given the customer is this and the order quantity is that and the product is this and da, da, da, then the system will generate these outcomes. So scenarios and outcomes come in pairs, in effect. So, okay, that's O. Now onto F, O, S. >> S. It was S. >> P is prediction.
Now the thing is, if you said I've read your requirement, let's say you write a requirement and I read it and I think, okay, so what this requirement is telling me is if I do A, B, C, then the system will do D, E, F. Is that correct? I've made a prediction. And that's what -- that is the oracle, you know, that we use for testing. This is the source of knowledge that we use to predict expected results when we test. But it's also what the developers use to identify what they must build.
So what a requirement -- one of the purposes of a requirement is to act as an oracle, to allow us to predict what the system should do in every situation, ideally. Of course it can't do that because there are no perfect oracles. It would have to be a massive, massive, massive document, perfectly written in mathematical logic to have an oracle that was perfect. But it's always compensating and compromising the requirement to figure out what systems should do. So prediction is really important.
And just one of the challenges that we can have is, like, I can think of an outcome, but I can't see the outcome described in the requirement. But it's an outcome that's sensible. You know, a requirement might say the system will process orders, da-da-da, gives you all the rules, this, that, and the other, but it doesn't talk about invalid values. And as a tester, you'd think, well, hold on, I'm going to throw some crappy data at this system. What's it going to do? And it doesn't say anything.
It just says reject. Well, reject is one thing, but it might be more complicated than that. It might have a different pathway. It might be acceptable in some circumstances but not others. And if those rules aren't defined in the requirement, how do I test it? But how does the developer build it? So by acting like a tester and looking at requirements in a very critical way, we help everybody. We help the developers. We help the users get their requirements right.
And we help ourselves because then we have a better understanding of the requirement and how to test it. So prediction is really, really important. And I think we forget that. That is our oracle. Literally the requirement is our oracle or should be in an ideal world, in an ideal world. So that's O-P-A is ambiguity. So I'm up to A now in the Defo span. You'll have to put Defo span somewhere in the video. Yes, we have to underline it. So A is ambiguity.
And ambiguity is -- there are layers of ambiguity. So there's ambiguity in the language we use in that you might use the word customer. Well, customers could be new customers or old customers. They could be high-value customers or low-value customers. They could be consumer-level customers. They could be corporate customers and so on. So when you use the word customer, should we not qualify it to make sure everyone understands where we're coming from? It's ambiguous. It's an ambiguous term.
Without some reference, some qualification, it's an ambiguous term. There's that level of ambiguity. But there's also ambiguity whereby it may be that there are two scenarios which look very different which generate the same outcome. Now, that might be okay. But then it might not be. Because the first scenario is written on page 23 and the second scenario is on page 38. And so no one has made the connection that these two things are actually ambiguous.
There should be two different outcomes, not the same outcome. Or the other way around, you have a requirement stated on one page and another requirement stated on another page, and they are different scenarios. Sorry, they are different scenarios. Sorry, this is not right. They are the same scenario. Sorry, I need to get this right. They are the same scenario, but they generate different outcomes depending on which part of the document you look at. So which is it? It can't be both.
So a tester would say, "Oh, this is ambiguity." Okay, so I'm not explaining that very well. But you get the idea that we don't find it easy to check these things. And yet if we do, we usually find some really good problems in the requirements, which if we don't find them then, we don't discover them until later on when someone says, "Well, page 23 says the system should do this, and it does something different." "Oh, but page 43 says it's working perfectly."
"Yeah, but this page says the same thing, but it's a different scenario." It's the same scenario, but it's a different prediction. And in order to get all that inside in your head, you have to work really hard as a human. But the investment might be worth it. The investment of time might be worth it. That's true. So that's ambiguity. Last thing is missing. So, you know, are there missing scenarios? That's an obvious one.
It's very easy to spot, "Oh, what about this combination of inputs?" You know, that doesn't appear anywhere. How do we judge that? There might be missing pieces of data. "Well, surely the age of the user would be relevant in this situation." "Oh, yeah, we don't talk about that." "Well, but we should. It should go into the requirement," and so on. So missing aspects of requirements, I mean, the world's your oyster.
I mean, it never ends because requirements usually, they cannot predict every outcome of every eventuality because the document would be just massive, and it would just be an overwhelming task to try and do that. However, the most important things that are missing are still the most important. So we need to pick those up. So we need to be selective. But that's it, the whole thing, defo spam end to end. And this was in the book about 12 years ago.
And people said, "Oh, that's very interesting," but I never met anyone who said they'd used it. So it kind of got put on the shelf and forgotten about. And just before the retreat, when we met two or three weeks ago now, I thought maybe AI could help with this. And that's kind of where I'm at in that I'm thinking about that now. And I've begun some more serious work on that.
So I think AI can help us to improve our requirements and generate stories, perhaps, or given stories, it can analyze those, you know, Gherkin-style stories. But also find big holes, you know, and it can help us with definitions. It can help with predicting outcomes. And AI seems the way out when I started looking at this as a challenge and actually putting some requirements into chat GBT, needless to say, it didn't come out with scenarios and outcomes.
It just came out with scenario outcome pairs. Because, like, it almost doesn't make any sense to have an outcome that doesn't have a cause. You know, that's the end of the sentence kind of thing. The outcome is if this, then that. So we've got the that, but we also need the this. So we have scenario outcomes as pairs. And that seems to be how the AI likes to work. So that's fine. So it's not two stages. It's one stage now, I think. And so on. So if you want me to talk about that, I'm happy to.
But I've talked nonstop for about ten minutes now. [Laughter] >> I think it's very interesting. What are your experience so far now if you're using AI for this, for your definition part, for the functions, for the outcomes? What are your experiences up to now? >> Well, it's a bit of research. Let's call it that. Okay? I'm going to write some code. I'm going to build a prototype because I think that's the way to explore its capabilities. And I think it's kind of promising.
I don't know if there's money in it. Don't get me wrong. It would be lovely to make millions and millions. But that's not really -- >> Like everybody. >> -- on the table. I'm just having a play. But I think it could be really useful. Certainly if you ask AI to take some text and identify all the terms, you know, named entities is the phrase, things that need -- things that, you know, nouns in effect or actions or activities, named entities, it can identify them very easily.
It can also, given the context of the documents you feed it, come up with reasonable definitions. It may not be perfect, but it can get you started at least. So definitely can help with definitions. It can identify scenario and outcome pairs and match them and saying -- it will just -- you know, if you ask -- if you give it a prompt that specifies the output you want and the style you want, it will generate these pairs. Now, what it will do is it will not generate very many.
So what we think is a great requirement typically is very low on content. There's a lot of implicit assumed requirements that, yeah, of course it's going to do that. Yeah, but, ah, you'd have to write that down. Well, AI doesn't know that. So it will give you a relatively small number of scenario outcome pairs. And so the next obvious question is, well, can you think of any other outcomes, scenarios, scenario outcome pairs, whichever way you play it, and it can do it.
And it will -- because it will -- you know, if you're talking about order process, the example we used at the workshop last week or the week before, it's an order -- it's an online e-commerce application. We used it as an example. We have a set of requirements, which are, I think, deliberately poor, but whatever. And it knows what an e-commerce site should do. So it knows that there are boundaries in terms of how many of a certain thing you can order before you run out of stock.
It knows about pricing and calculations. It knows about discounts. And, you know, it actually knows an awful, awful lot of stuff. Now, a lot of it will not be relevant, but you can ask it to add scenario outcome pairs to your requirements that it thinks might be helpful. And mostly they are. And they -- you know, so you start with, like, five scenario outcome pairs, which doesn't sound that many, and it isn't.
It will say, oh, here are another eight straightaway without even, you know, troubling itself. It's like, oh, crikey, yeah, and they're all sensible. They all make sense. Well, mostly they make sense. So you think, oh, we could add those in. So you're increasing the richness of the examples you've got. What else? Ambiguity. You can ask it a straight question. You can say, can I look for ambiguities of language, of named entities, of inconsistent scenario outcome pairs.
And it will give you a list of things it sees. And we can talk about whether we trust it. Yeah, we should talk about that. But I'm just doing the sales pitch now. Missing stuff. Yeah, it will add missing stuff. You know, it will say, perhaps you should consider these situations, these items of data which might be relevant in order processing, say, and so on. So it can come up with lots of ideas.
And in that way, it's a really helpful -- it could be a really helpful assistant to embellish requirements, to give you sufficient examples to fully illustrate this application. I shouldn't use the word fully. To illustrate, not fully. I mean, that's a moot point. It can't do that, I don't think. But it can give you lots more examples, which is great.
Because when you get around the table and you're discussing the requirements, you want examples to challenge the users and saying, have you thought of this example, that example, whatever. And an example is really just a scenario outcome pair. You know, if this, then that. That's all an example really is. It might look more like a test case with more detail. But that's all an example is. And examples are what help people to understand the challenge they've got with requirements.
Oh, it's more complicated than we thought. We need to add -- there's more fields of data we need to capture, more fields of data we need to validate, more fields of data we need to use to affect the processing of -- or order processing, say. I think there's a lot of potential in this. Because when I think about the missing part, so we often can't imagine what's missing. We have to think about it, what can be missing, but we don't see it because it's missed.
And I think AI can give us an inspiration. You might think, oh, from experience, I know this is a situation we should deal with because I'm experienced. But your experience isn't global and total and comprehensive and never-lasting and universal. It's none of those things. There's a chance you get a better, more comprehensive set of examples. I would use the word examples. The experience of AI, given what it's trained on, is probably greater than anyone in the team. Now, don't get me wrong.
It's not the best member of the team. It just has a much better memory. So it's an idiot savant. It remembers everything. It can do calculations like that. Yeah, that's fantastic. It can't debate. It can't discuss. It can't challenge in some ways, although I think you can encourage it to do some of these things a little bit more. But the thing is, it's like it's an idea generator. It's great. I mean, it's great for that. It's not comprehensive. Can you trust that its ideas are sound?
No, you've got to check them. You've got to keep an eye on what it does and get value where you can. It might do 80% of what you could do as a team if you worked a little bit harder. So it moves you forward but doesn't solve the whole problem. So is it trustworthy? Well, the short answer is no. Is it helpful? The short answer is yes. So it's an imperfect assistant. It's not going to -- would you recognize the code it generates? Would you trust the code it generates?
Not many people would say yes. And that's where you're at. But it might generate 80% of the code, and if it's structured according to your instructions, you know what that code does and how it does it. So you don't have to learn how it works, but you still have to test it, you know, or at least inspect it. But I think you're going to test it anyway, aren't you?
So I think going to step you back a little bit is the experiments I've run so far have been really promising with a caveat that it is what it is. It's not a genius that knows everything and can solve all your problems. That's not the answer.
And something I've been thinking about, which I think I've talked a little bit about in the retreat, was do we treat AI as a mentor in that it knows everything, can answer any question, and will help get us out of a hole, or do we treat it like a coach and it makes suggestions and asks interesting questions to make us think a bit more deeply? I think we should look at AI much more as a coach than a mentor.
So it doesn't have to understand everything about everything, but it's going to have some good questions that it can ask you to get you thinking about issues that could be really important. So that's the way I would pitch these things is that it's got a memory that you haven't got, and sure, it can come up with suggestions that might take you a long time to make yourself, and it might come up with some stupid suggestions. Well, which human being doesn't come up with stupid suggestions sometimes?
It's human in its failures. I mean, it's not a human. It's software, but it makes mistakes. It gets above itself. It hallucinates. It sometimes goes off. It misunderstands your question and answers a different question. Yeah. You have to keep an eye on that. You have to supervise it. But I think there's value in what it can do, and I think that value is worth looking further into, and so that's what I'm doing. I'm going to try and build some kind of prototype.
I don't know what it will look like yet, but my experiments have been kind of interesting. Yeah, yeah. Yeah, it sounds great. So I hope we can, in some months, we can do another episode where you can present a prototype too. I'll give it a test. We'll try it out. Yeah, it would be great to see the progress here. Yeah, Paul, thank you very much to be on the show here and to give us the insights in one part on your acronym. We have to spell it in the show notes so that everybody understands it.
And on the idea, how can AI support there and give us ideas and inspiration to make better requirements and to think more about requirements and what we missed and what is ambitious and so on. So thank you very much that you have the time here on the show with me, and I hope we'll see again latest on the next retreat. Well, definitely. I definitely want to demonstrate something or tell you it doesn't work. It will never work. I mean, I don't think that's the outcome, but it's possible.
I'm still open-minded. We won't hope that. So, yeah, thank you very much and have a good day. And you too. Thank you so much for having me. Thank you.