Hey, a quick message, for those of you who are listening to this episode on Spotify, I have a small favor to ask Spotify. Now allows mobile users to rate podcast, I would really appreciate it. If you can take a quick, pause to go to the technique Journal podcast page, and leave your favorite show. Your best rating on Spotify. It will help me a lot to get this podcast to reach more people on the platform. Thanks a lot.
I'm going to take this from Elizabeth Hendrick since one of my favorite quotes testing is an activity, that That happens throughout. It is not a phase that happens at the end. So if we truly believe that testing is an activity that happens in the very beginning. When we first see that feature, start thinking about the risks at the very beginning and start thinking about, how are we going to mitigate that? And a lot of that litigation
involves testing Hey everyone. My name is Henry Surya with Robin. And you're listening to the technology, you know, podcast the show where I'll be bringing you the greatest technical leaders practitioners and thought leaders in the industry to discuss about their Journey ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our Journal.
Hello to you, my friends and my listeners out there, welcome to the technology. Now, podcast the show where you can learn about technical leadership and Excellence from my conversations. With great thought, leaders out there, and welcome to the episode number 92. Thanks for tuning in and listening to this episode. If you're new to technology, you know, don't forget to subscribe and follow the show on your podcast app and social media on LinkedIn. Twitter ER and Instagram.
And if anyone wants to contribute to the creation of this podcast, support me by subscribing as a patron at technology. No, dot f /. Patron testing is a critical part of any software and product development. However, there are still many teams that consider testing as an afterthought or even making quality and testing. As the responsibility of Silo teams who are quite detached from the product development
life cycle. I've been wanting to invite guests who can share About how we can do better testing as a whole. And I'm so excited to share this conversation with you. All my guests.
For today's episode are the lovely duel Janet Gregory and Lisa Crispin Janet and Lisa are the causes of several books on HR testing and the cofounders of agile testing fellowship and they are very well-known and influential champions for better software testing approaches in this episode Janet and Lisa shared the agile testing concept and mindset with an m. Emphasis on the whole team approach.
Our discussion was then quickly followed by an explanation of the holistic testing concept with a complete walkthrough, how we can use the holistic testing approach in our product development cycle, including how continuous delivery fits into holistic, testing Janet, and Lisa also describe some important Concepts in agile, testing such as the age are testing quadrants that we can use to help classify and think about variation of test cases and the Power of three also
widely known as The Three Amigos towards the end. Janet, and Lisa, also shared their insightful perspective on conducting better exploratory, testing and demystifying, testing and production. I really enjoyed my conversation with Janet and Lisa learning about the concept of a jar and holistic testing and how those approaches can help us to create better software and product. I especially like their Dynamics in our conversation which made the discussion Lively.
If you also enjoy this episode, please share it with your friends and colleagues who can also benefit from listening to this episode. Leave a rating and review on your podcast app and share your comments or feedback about this episode on social media. It is my ultimate mission to make this podcast available to more people. And I need your help to support me towards fulfilling my mission. Before we continue to the conversation. Let's hear some words from our
sponsor. Today's episode is At least sponsored by skills matter. The global community and events platform with more than 100,000 software professionals here members. Can organize their learning experiences around the technology topics. They care about most you get on-demand access to their latest content thought, leadership insights, as well as the exciting schedule of tech events running across all time zones.
So whether devops our data science is your bus or you are fan of functional programming or all. All things Cloud you can make real connections with people who share your interests head on over to skills method or calm to become part of the tech community that matters most to you. It's free to join and you will find it easy to keep up with the latest tech Trends. Are you looking for a new cool swag taglit Journal?
Now offers you some swags that you can purchase online these wax are printed on demand based on your preference and will be delivered safely to you all over the world where shipping is Available. Check out all the cool swag is available by visiting technology. You know that death / shop and don't forget to break yourself. Once you receive any of those threats, hello everyone. Welcome back to another new episode of the tekhelet. You know, podcast today. I'm very excited to have the
testing Duo very much. Well, known in the software industry. We have Janet Gregory and Lisa Crispin in this episode. So, for those of you who don't know, both of them, they are actually the author of a gel. Testing book which was published in maybe 2009. So it was kind of like a revolutionary during that time when all a gel is in the bus and also try to improve testing practices and they wrote another book, which is titled more agile testing, this was published in 2014.
Maybe there was something that they want to improve from the first book. And recently, they also publish a book which is a kind of condensed version of both books called agile testing condensed. So as you can tell today we are going to talk a lot about a gel testing and some Testing related stuffs for sure. So Janet and Lisa really pleased to have you in the show. Welcome to the technology, not podcast, thank you. Thanks. It's really nice to be here. Thanks for inviting us.
I always like to start with asking my guest to introduce themselves. So maybe, if you can take turns and introduce yourself telling us more about your highlights and turning points that were to share. So, I had a little bit of a different start because I worked right out of high school and then I got married.
We'd had kids did all of those things, did some traveling lived overseas for a couple of years, Singapore, in Jakarta, and then when we come back I decided I needed to do something with my life. So I went to University and I graduated computer science and became a programmer but this was all around the age of 40. And then I started programming for about, I guess, six years that I like to tell people, I saw the light and I went into the testing.
Was a QA manager for a few years, but my first job was not very satisfying because it was in a very chaotic kind of environment. Being a QA manager, all about process and nobody was listening to me, and I was so frustrated. And when I quit that job, I was quite lucky because I ended up in an agile team as a tester. I just kind of washed my hands and started all over again. That really truly did change my life because that's when I met Lisa.
Sup, because I had this moment of panic thinking what does testing in an agile? I don't know. So I met her because she was writing her, very first book testing XP. So I had the opportunity to help review that book and learn lots. And then, of course, that Lisa and I have had our own kind of separate careers, what very much bound together. All the way through I was tester / coach because I was one of the few people that actually understood how to test in an agile team.
Until our first book was written, I should say. And then I started Mark Consulting and then I've been Consulting ever since Lilly helping teams to understand what it means. So that's the my career in a nutshell. When I went to college you couldn't get a computer science degree when the thing you had to get electrical engineering, but my original degree is in animal science with a focus in beef, cattle production and then I got a Masters in Business Administration and was had a
research job. But at some point I needed a job and wandered into the University of Texas at Austin employment office, and saw a sign that said, programmer trainees needed, no experience necessary. And I said, well, that's me and I'm fortunate that a, they hired me and be, they hired me for my
domain knowledge. They want to people with disis background, they had a great idea, the train programmers themselves so that we all could have Collective code ownership because we all coated this, a way we work together. We collaborated that was back in the day when programming was a low pay low status job. So there were plenty of women. A lot of fun eventually. I went to work for a software vendor and kind of accidentally as most people do got into
testing. Oh, our customers are really upset that all these bugs go out with our releases. What if we tested it? And so that was in the early 90s. When I switch from programming to testing, I've just been fortunate. I've just been on so many great teams. Most of the time. In my career, I've been on some bad ones but mostly great ones, I was able to jump my first extreme programming payment 2000. It was difficult for testers because it got all the publication's of the time about extreme.
Program, maybe they were great. It was all about quality is all about testing. It was all about people but nothing about testers and what they might do on an extreme programming team. So you know, my team and I were trying to figure it out through the extreme programming, mailing list, I met Janet. So we were all trying to figure this out. Together Co wrote my first book would tip house because we want to share what we learned so far. Because we knew there were other challenges.
So it's been all about sharing experiences. And I've been very fortunate to collaborate with Janet since then not only on the books. But on conference time Since and read other articles and training courses. It's just been great to reach out to our software. Can be maybe testing Community around the world, get everybody to share experiences, and we kind of channel that out and share it to everybody as much as we can. Thank you for sharing both of your stories if I can pick some
of the interesting things. So I didn't know you have stayed in Singapore and Indonesia China lesion by the way. So that's kind of like similarity from both of us. So, both of you studied programming but then went into testing. So tell us a little bit more white Testing, apart from so many other areas in software, engineering. Okay, my very first job in programming. I work for the Vancouver Stock Exchange and so things had to work.
I was very fortunate that my boss every day would come in and sit with me and say, so, Janet, what are you working on today? There was no pairing. We all set with cubicles and things. So I would tell him what I was going to be working on. And then he'd say, so how are you going to test that and in at the end of the day, he would come. And you'd say, so Janet, what did you do today? How did you test that? So I learned to think about testing right from the very
beginning for my first job. And I didn't know any difference. So, when I went to my next job which didn't do testing on their things and was working out just as critical application as Stock Exchange, it bothered me. I coded for a couple of years there and then my boss come to me and said so Janet's. We think we need to have a test team. Mmmmm. You are the only one who was constantly complaining about our princess. I would like to be our QA manager. I knew that I didn't want to
program forever. I knew that from the very beginning and so I actually jumped at the chance and then I took everything I possibly could learn about testing and started there, so it ended up not working out for me there, but that's how I got into it. I've never looked back. That's kind of like very interesting question. How do you test that? Because I think many software Engineers even don't think about that. So they just build the features and finish and maybe pass it to
Q&A. So, how about you Lisa? Do you have any interesting insights, why you went into testing world as well? That's a pretty good story. So I was working in tech support for a software vendor a pretty big software. Vendor, this is back in the days when Microsoft and Apple were still small. I think we were bigger, we're frustrated because that was back. When you sent everything out on paper to the customers, These are Big customers that was
mostly with database software. We do a release, the tapes would be sent out. We had nothing to do with it. This was back. When people just called on the phone, no faxes. No email. Just had an angry customer on the phone. Who said, I cannot believe you didn't see this giant bug. We hadn't even seen it. We didn't have it yet either. So it occurred to us to ask the developers. We were in Colorado and they were back in Germany.
And we said, could it be possible to send us maybe a preliminary version before you send it to the customers? And we could just Start trying it out. So they said, yeah, we could send it like a week ahead, so we could install it, we can start testing it, we could find bags and we can start working on patches so that when they angry customers called basic, don't worry.
We're working on a patch and we'll have that out to you next week and we did it on our own and then our managers were like Sting, what an interesting idea. What if we have a department devoted to people who do testy and coordinate their releases and cut the released tapes and they will the customer Maybe we should combine all these things and so I put my hand up like that, sounds fun, and like jamming.
I've never looked back. Wow, so both of you took the chance jam into the opportunity and went ahead an till now. So which brought you into the concept of a gel testing. The first book was written in 2009 if I'm not wrong. So what is agile testing? Maybe if you can give the definition for all of us here who maybe have not heard about
it yet? Our official definition collaborative, testing practices that occur continuously from inception to delivery and Beyond supporting frequent delivery for our customers testing activities focus on building quality into the product using fast feedback loops to validate our understanding in the practices, strengthen and support, the idea of whole team responsibility for Quality, we spend a lot of time trying to put that together but really what it means is play
nice in the sandbox. Work together, think about testing from the very beginning when I think about this and we might talk about a little bit later is that definition could be applied to the term were using these days, which is holistic testing. And by the way, that was a
community effort. We got a lot of inputs and people in the community, we've written a couple blog posts on as we were developing that and so you get more details on our website at altestore.com slash blog and search a room for that. But yes, Janet that's a really good point that the definitions don't. Works, even if we use a different label. So if I read the descriptions, definitely first, it's like, there are so many things mention there. But there are a few key things
that I observed. The first thing is to mention about this concept called whole team. What do you mean by whole team? What is the relation with testing as a whole team and our experience being successful at delivering valuable software products to customers is an effort that requires everybody on the team to be committed to
it and work together. We've seen over the years that Saying at the end only by tester doesn't work, even when I work in waterfall projects, everybody still did testing the whole time. Wonderful doesn't mean you have to do a bad job and having that mindset where we've all talked about that everybody on the team. Regardless of our specialty, here's a global equality. We want our software. This is what we want to get
through. This is our goal and we're committed to getting there because we know, it'll be hard. We know where run into a lot of obstacles, will have a lot of hard problems to solve but together with all our different skills, And experiences will be able to do it because we're committed to it. We're not going to just throw our hands up and give up. That's what it takes. If it's just part of the team, even if some of the developers like, oh yeah, I really care about quality.
Alright, some unit tests, but if that's all they do, and they don't involve. But testers or the testers go off and try to do all the testing on their own. We just seen it doesn't work. The state of devops survey has really supported that with hard data that was damaged in my experience over all the years. But then we saw data from that survey supports It when developers own the testing, when they own the automated test, yet they created them and maintain
them along with the testers. And the testers help them with all these other activities, like exploratory kissing. That's what correlates with high performing teams. I know as humans we don't pay attention necessarily the hard data but we have the hard data to back up our views and it starts from the very beginning. That question. How are we going to test this? If we start that at the very beginning that drives the testing all the way through.
So I think I agree. It's a very important distinction, having the whole team, really taking responsibility of how to test the application or the system that they're building. Because as you can tell in the industry, I'm sure throughout your Consulting experience as well. There are many teams still have the silo responsibility developer just build the
software. While there's another team called the testing team, QA team, quality engineering, whatever you call it. That's the testing part. So what in your opinion should change in order to make it more like a whole team concept, all this Still kind of like a common Trend in the industry. I can tell at least from my part of the world. What should change in order to go towards this whole team concept.
Well and I'm going to take this from Elizabeth Hendrick since one of my favorite quotes testing is an activity that happens throughout. It is not a phase that happens at the end. So if we truly believe that testing is an activity that happens from the very beginning. When we first see that feature that very first feature starting to think about what are the risks X to be able to do that. That means that who's ever thinking about testers.
If you have testers on the team, if you don't have testers on the team, start thinking about the risks at the very beginning and start thinking about, how are we going to mitigate that? And a lot of that mitigation involves testing it starts there. And so moving, both testing activities, thinking about the early, I think that's how we're going to change it, don't bring in testers when you've got code
to test. The man early to start thinking about those risks and really talking about that level of quality because that's how we start it. And I think that's has to change its the mindsets of fatigue and individuals because it's how we think about testing. So I don't use software tester. I don't use that term for myself because I think it's more than software. We test ideas, we test assumptions. We're testing many things. And so it's not only the software. I think we test the Product and
the product is the whole. Yep, which you mention is, like a mindset, right? So in the book, you mentioned agile, testing mindset. So I can also see, observation from at least my part of the world. So testing is always the Miss. Like, maybe lower part of engineering, right? Where people assume they just do manual testing during the reproductive jobs. I know that some testers are like good programmers as well.
They're right great Automation and all that but still unfortunately, there are still these misconceptions also This term called quality engineering quality assurance, so people think they added efficient that is responsible for Quality. So I think all these, maybe need your advice, how to change this mindset, so that we can move towards this, a gel testing mindset. Again, where everyone will involve in the testing and really cares about quality since
the beginning. Yeah, Ted. And I were just chatting about that this morning, because it's not easy, and I'm just trying to think back that had in mind, mindset changed. I remember the first first two-week iteration on my first extreme programming team. I was working for a start-up consulting company and most of the first two weeks, I was helping another client of sight. So I came back just like the day before we're going to demo to the customers or a very first iteration.
One of the things I did was okay, I'm going to start up the server and we got this little application going and I'm going to log in as two different users and the server crash. And I was like, what? We can't show this to the customers. It's terrible. You can even have two people on it. Fortunately, an extreme programming. You have a coach part of your team. Archive said. Now. Lisa, we don't have a story in this iteration for supporting
more than one user. The reason for that is our customer is developing this to show people the potential futures of their product. They need funding and I'm going to use it, they're just going to demonstrate it. They don't need to have two people logged in. I was like what, what what oh that's mind-blowing to me. Quality was the server says up its reliable. Thing works. But now the quality is what the
customer needs. We really had to focus on that so I really think it's an ongoing process, I think it needs training. I think it needs daily coaching from somebody who's done it before, who knows what they're doing to help the team Janet. And I like to say, if you haven't already been on a high performing, a dollar whole team
approach. Team, you can understand what the unicorn magic of that is, you really have to experience it. And so, you know, if people can get somebody hire somebody at least temporarily, Like who does know, is who does understand it and can help the team get over it, because mindset switches are hard. It's a cultural change. As Janet says, you know, we're going to focus now on preventing bugs not catching bugs at the
end. We're not going to be the quality police because it's not our job to determine what quality is. It's the customers now we've got to find out. What does the customer want? That's big part of our job. As testers this, to help everybody understand, get a chair and understanding of what the customer really needs. That's a really good story Lisa.
I wish I would have heard that 20 years ago because That's exactly it. From a testing perspective, the hardest thing is to understand that we are testing this small slice. We are making sure that this small slice works right now, doesn't mean that we're going to give it to the customer right now if that's what they need, if they can use it. Yes.
But from a testing perspective that is the hardest thing is to understand that their testing is narrow for this particular Story. Once testers get used to that they realize Much easier it is to keep testing and adding complexity. You have that first story and then that works. You add complexity and you can test that and that works. You just keep wrapping that complexity around and you get a solid feature. And I think that makes the world of difference.
But it is a hard mindset to think that we have to do that narrow. It's funny. I have to laugh here. I just going to add a little story because I can't talk without my hands. So, and Gasps, you have to visualize that I'm holding my hands up and I'm doing narrow Focus.
That's true. That's the thing in my early days or even for the first two years that I was on agile teams, my teammates had to keep reminding me, Lisa, I know you just think you found a bug could you just make sure the happy path works first so that we can be confident that we have at least started in a good way because, you know, I was always going off in the weeds. I bet there's a bug over here. No. Lisa, we think it works.
We want you to help us. I know that it works driving development with those tests, I was like, creating huge numbers. I was with the product owner, we're creating these huge matrices of test cases and expected outputs and given that the developers, okay, here's what you should do. And they're like, oh my gosh, I can't see the forest for the trees. So can you just give us the happy path? Let's all make sure that works.
Then let's add on like one unusual scenario and just do the step at a time and that was something I had to learn because I had been more used to even though I was trying to always collaborate with Developers and Tessa still, we were doing big, bang things. So I was getting a whole bunch of stuff to test that once and that's a quite a different
experience. Yeah. And I think when I've taught a course hundreds of times and I think that's the biggest takeaway most of the time from the whole team because we teach it to the whole team is that they can do that once. Well piece and add complexity as they go and test it. And I think that's a real showstopper for a lot of people just going. Whoa. And she said, I was laughing when you shared that because I wasn't even Being about application, that should just work for one people.
We always assume that it could work for multiple people. So you brought a point where sometimes you have testing is hard because you don't know what kind of slice that you should be focusing on. And I think you mentioned the key point where we should test within a smaller scope and complexity afterwards. And it seems like this kind of like a line to your concept of holistic testing where you have this short cycle from build test deploy and all that.
So maybe this is a good time to actually introduce. What do you mean by a holistic? Testing that You are talking pretty lately. I'll take this one, because there's a lot of talk about shift left shift, right? And when I hear shift, left shift, right? I think of a lateral line, like a horizontal line, I'm thinking that software development isn't like that. So the devops loop started and I think that's great for some I had seen that was in discovered to deliver Island Goddess Diana
and Mary Gorman's book. They had this nice little loop, the devops loop, and then Dan Ashby, and one of his blog post talked about Newest testing, he said the step up so it was great. But where does testing? They always had this choke phase for testing and it didn't work. So he made this we test here and he put all around the loop. We test here and here and here and everywhere. And I thought, perfect and I use that for a long time but still something bothered me about the devops loop.
So Lisa, come up with one iteration and then we just kept playing with it. And finally, I kind of had this little mini light bulb moment and took the the loop as it is now, Now and wrote a blog post on it showing that testing really does start from the discovery.
So when we start thinking about Discovery, that's a very beginning thinking about the risks and planning for it going through the building and those are all stages that we do, whether we go really fast through them because we're doing a small story, we discover we understand and we deploy and test, we might put it into production but one way or the other, we have an internal release it.
And if we don't put it too, Action and if they gave out what we learn from it. So it really is looking at that whole cycle, thinking about testing holistically, when I read that definition, and I hadn't really thought about that definition agile, testing, and it's very applicable to holistic testing. But I'm going to let Lisa talk about why we call. It holistic, testing versus agile testing these days. I mean, obviously, it was Janet
that came up with this idea. But part of the problem was Dan Ashe. We use continuous testing, which made a lot of sense to us. And then somehow people took the term continuous testing and co-opted it to only mean the automated regression tests that run continuous integration. And so when you s taste testing, that's what people thought of it was like that's like a teeny tiny little part. At that thing, it's an important part.
What describes it better. So Jenna came up with the word holistic, and, yeah, because whole team approach going through the whole cycle. The testing activities are in the whole cycle and it really sums it up. What I found interesting, isn't it? Just recently, I was still a hint on the tester on a future team and we're going to company with several different teams. I shared Janet's blog post on our slack.
One of the really experienced testers on another team was like and he had a light bulb moment to, he says, this model is great. I can finally explain to people what it is. We do because it really summed up when there are well, performing teams that are doing a good job of the holistic approach. And there's a tester on that team. Kind of acting is a testing consultant. Guiding them. Leading that effort now, he can
take it explained. The other thing, this is what we do. I have a way to show you now, other people have also come back with that kind of feedback. One of the ways that resonated with people who are already doing it, recognize it. So, we hope it'll help other people than be able to achieve that holistic approach in grow that in their own teams. So maybe if you can walk us through in terms of practical sense, right? So you have this cycle, there are multiple stages in the
cycle. So I just pick a starting point Discovery as you mentioned that you have planned. Understanding building deploying releasing observability and learning throughout these stages. There should be some level of testing involved, so maybe if you can walk us through in Practical terms day to day, maybe in the spring, how do you run this? Do we create a story for each of the stage, or do we have one story where each has its own stage.
So if we think about it, say The Discovery, I would say we have a story. Let's take the simplest story that we possibly can one person can log in using leases. I think so that story is the happy path. Lisa can log into the system, what does that take? And so we would take and we would think about what risks there are. Maybe the risk is a servers and up, she can't get validated, whatever it is. We think about the risks. We might share some acceptance tests. High-level acceptance test.
We might do example mapping if it needed to be, so that's part of the understanding. So you're planning at a high level. What is it take? You understand that. All right. And every story will be different depending on how well loan it is. What are the risks? So, then you have some high-level acceptance tests. You were two examples and the whole team has an understanding of what they are going to build and then they build it. Of course, the building will include automating it.
Include exploratory, testing on that story and includes tdd for the programmers. It includes whatever it takes to make that code into a Deployable piece. So you have that code. Deploy it your automated tests are already running. You can run them on your system. You can do any other human Centric, kinds of testing, like perhaps you have to do some performance testing or something
like that on a deployed system. So all of these little testing activities happen and then if you really sit with us internally or externally to a customer, you're going to make sure that happens and so testing in production, every time I say, testing and production is that back 20 years ago? No, no That's not what we mean anymore. But you can do that. And then you can have your observability and you're monitoring happening, but that's all about learning. How is your customer using it.
If you've actually put it out to production if they're using it and then you learn and you go into your next story. So that cycle can be very fast. If you doing something very simple like logging in with a valid user, we're not doing any extra step, we're just doing that happy path. The next story might be validating all.
The usernames adding complexity. Maybe the third story is something else, but you adding complexity all the way along and I think that is recognizing that Loop is just a continuous cycle constantly adding on. One thing I particularly like about the loop is haven't this part where we're going to learn observing production that's one of these past few years.
My mission has been to make sure that testers get involved with that because our products are so complex these days were Is the cloud we're using all these Services microservices we need to start as we're thinking about what code we're going to rate for that little story that Jan has talking about, what Telemetry do we need? How should we instrument that
code? What events do we need to capture so that we can create our monitoring dashboard so that we can create our alerts, if something goes wrong in production. So we have all the data we need for the things, we didn't anticipate and didn't create dashboards or alerts for, and we can learn very quickly what went wrong and fixed it and also we're We're so lucky these days.
The analytics tools that are out there which we're done for marketing and product so they know what customers are doing. So valuable to us as testers, we could go out and look at Sea, power integers, using our product, you know, some of their tools and I'll show you a screen cast that what somebody did, which is kind of creepy. But it's very educational when you see somebody clicking on something that's not clickable or just moving the mouse around because they don't know what
they're doing this. So helpful, to improve our product to know where to focus. Our testing to know how to help you. Hers, we need the whole picture. We can't just be people say, shift left shift, right? We can do one of the other. We've got to be holistic. We've got to go through the whole cycle and be involved all the time. Using our testing skills to identify risks. It's a spot anomalies to spot
patterns to help the theme. See those, I was just actually listening to your last podcast with Liz found Jones. She was talking about observability, one of the things that she said and I went this is the way to think about it because it hit home. The way she said it was, can you understand what's happening in your system and why without having to push additional code slicing and dicing data from the Telemetry signals. And I thought, what a wonderful statement that is about observability.
It sometimes I well, because I remember somebody paid for years of oh God, we cannot figure out why it says 500 error happening with our web server. Okay, well let's add this Telemetry and let's redeploy. A little that didn't do it. Let's have this very good boy. No, that lives, this one of my sheroes, the people in that space and in observability and site reliability, engineering. They understand testing, and they understand the importance of having this data.
We all need to be aware of that. And not think that some separate area of expertise that we don't need to know about. Lisa said, that's one of the testing activities that happen in that understanding is what do we have to put into the coat? Having heard what, you just described both of you in detail in depth including an example how we can practice it? I think It totally makes sense. I can see why in the industry, sometimes we all work in silos, most of the times developer just
read, okay? This is the story requirement. That's it. I don't think about Telemetry, I don't think about how we can measure user success and all that. I just built based on the specification has this probably also follow. So, I think the whole concept of having this cycle for every story, you can be bigger scope. It can be small scope.
But having all these cycle covered in a story actually, really good, because you can have a good understanding in depth from the start from the discovery up. That use that themselves how we can monitor and make sure that the story itself brings value to the user. Thank you so much for explaining this concept.
One part that I think with the mention because you assume that the story can be deployed to production where you have the release deploy and observability but so many people still haven't practiced this, which is continuous delivery. How important is continuous delivery.
In this holistic testing, well I see so many different contexts to make continuous delivery is Is something to think about and to strive for, even if you live or do it, a lot of things are on three months, release Cycles, delivering to the customer between month, and it might be because their customer chooses, not to take it. It's a requirement that a lot of
customers do, they? Don't want almost little fixes for many reasons, but if you have that goal and you're thinking about it, then that means that your stories are going to be testable. Your stories are going to be small enough. You can test and you will get
those good habits. Even if you choose only to give it to the customer who wants to. Well I encourage teams to practice continuous delivery to go for that even if it's on an internal server and they're practicing it on their staging environment, they can still get to that even if they don't push it to the customer every time, it's a great way to get all of those practices in place. So it's not going to happen overnight because a lot of things Don't even have
automation yet. How did you get there? That's just a series of small steps understanding. That every team has a different context. So we talked about it as everybody does but they don't they don't there are still teams out there that don't even have continuous integration. I remember when I told Lisa that was probably 10 years ago or 15 years ago and she goes what? Because every team she worked on problems and she couldn't see
working any other way. But they're still teams without continuous integration, which, you know, it's kind of sad, but I know they exist. It's puzzling to me because like, it says everything I've been on. I had it because it only took us a day or two to get it going.
So, what is the problem? People, it's not hard, but I like Janet says, before we even had the words continuous delivery, which I first learned from Jazz humble, and they Farley's excellent book of the same name, which is a book about testing. My team, took us a few years because at first, we couldn't even get a passing build with an artifact. We could deploy to production. In two weeks. And then we tried to get one
everything we set goals, okay? We're going to have a passing build every day and for the days we don't have that. We're going to have a calendar where we put colors to remind us. So everybody in the company can see including the executives and start to understand the importance of having this passing. Both everyday just build it a step by step. It took us years to get there but then we always had a
Deployable artifacts. We could deploy to production in our business domain customer didn't want new changes every day so we only did it every two weeks but we had the ability at, that's the goal. One thing I tell people that don't have anything, they don't even have CI, you have a deployment pipeline changes that you make in your software
product gets production. Somehow there are steps that you take to get there even if their manual and it's really important that the whole team understands all those steps. So we tell people sit down together, but use your virtual whiteboard and virtual sticky notes and draw your pipeline, even if it's manual, or even if it's some automation, some manual, understand what you do. And then try to pick the play sets. May be your biggest bottleneck where it would help you to automate it.
Get that faster feedback, make your process more robust so that you do have something solid you could get to production but you have to build a step-by-step. I learned from Janet years ago, the first step in solving. A problem is make it visible. So is your problem is you struggle to get anything to production because you're still have good processes.
Start by visualizing what it is now and then decide how you want to improve it. I will be surprised if many people Still, do not practice, continuous integration but few years back I could even see some teams do not even have Version Control or maybe their Version Control even just locally. So I think this is also a bad practice. I think we should all avoid. But by now, I hope all the listeners who have practiced at least Version Control and
continuous integration. So in continuous delivery, we know that you have many stages in your deployment pipeline. Obviously, many of those stages will be automation test. So I know that you have this concept agile testing quadrants where you have the garage. Has different types of tests. Maybe, can you give us an overview?
What is agile? Testing quadrants and what are the representations for each of the quadrants in actually wear red petticord in myself and Brian Merrick and it was one of those conference late night. Chats, we were talking about this and Brian was explaining these quadrants so he had a napkin it's one of those stories, right?
Say had a napkin he's drawing on it and then Brett and I booked him for months until he finally wrote it up in a blog post, so then we could use it. We did ask his permission to use it but the quadrants is a way of classifying, the tests four quadrants so the left side is about guiding development tests that you right before you start coding the right hand side is
about critiquing the product. So test that you do after coding is complete, the top bath is business facing tests, so those are tests that your business would be able to look at would be able to understand would be Able to do feedback on and the bottom half is technical facing, which just means that the language that those tests are in a something, the business wouldn't look at, for example, Performance testing, they care about the results, but they would never go look at the
actual tests. So the four quadrants, and they're numbered for ease of use only technology facing tests. That guide development. 101 and those are kind of unit tests. So the programmers are mostly responsible for guiding. Development with the unit test quadrant 2 is business facing tests that guy development. So this is where when we do acceptance test-driven development or behavioral driven development thinking about how are we going to use those
examples to guide development? What tests can we write before? So things like simulations or those sorts of things. Those are all things that help us quadrant 3 is business facing that critique the product. So generally we put in exploratory, Retesting in there or user acceptance, testing business facing tests. We need to do them because there's a notes, what didn't we think about? And then Quadrant 4, which was the quadrant that made me really
like this. Because at that time and that was in 2003 that conference, every time I would talk about testing in agile projects, every tester. I meant would say it won't work. They talk about customer tests and programmer tasks. But what about all those other? Chests. Quadrant 4 gives us a way to talk about all those other tests performance reliability in all the quality attributes that we know we have to work with. So the quadrants are a way of visualizing your testing.
Again, we encourage teams to think about the test, they do and then kind of put them in the quadrants to think about why are they doing them? Are they doing it to support? Because the left hand side supporting or guiding, that's about, About preventing defects in code and we test our assumptions. Can we test our knowledge before we ever write a line of code? The right hand side is about finding defects. What didn't we think about?
So it's a way of classifying it. So it's a way of visualizing the kinds of tests and where do we actually want to do this? Thanks for walking us through the Adele testing quadrant. So maybe for those of you who cannot visualize what Janet is saying just now, so maybe I'll put it in the show notes. So what the diagram looks like, But there are so many categories of tests. Depending on the left side, top bottom, maybe you can have a look.
But importantly, there are so many types of tests for one particular story should be built. So many of those tests, what is the ratio here? How should we build these test cases? Because I also notice that you have this good practice which is called The Three Amigos or the power of three. So maybe walk us through a little bit on how we should write all those tests and how I wish it ran all the tests and all the quadrants together with each feature.
Sure, that your team's going to work on and with each story that you work on, it depends for like a web application. Maybe you're doing something with a user interface and then it has a back-end server and the database have all those pieces, we often will start in that quarter to that business facing tested guide development, with prototypes and wireframes. And what is rui going to look like? We can test those, maybe get involved with designers.
When we say that the power of three these days, I think you're going to need four or five because you often need the designers or We have experts or aberrations, experts, it could be anybody. So we make sure that testing those feature ideas is Janet mentioned earlier. Once we've written sliced up this testable stories and start working on them down. In that technology phasing quadrant one that guides development. The developers are the programmers are coding test.
First are doing test-driven development at the unit level, that's going to be something that the programmer Zone because that's not a testing activity as much as it is a code. An activity, but it does help us design, testable code, and operable code, all those good things. Then we're going to maybe go back into that quadrant two and do our story testing. And again, it's great to have a programmer in a tester pair up on that or attachment a product
owner or even better. For my point of view and Ensemble with several people in different roles testing it together. Well, I guess you can do that. So we set the story Little know we were for exploring later on but Beverly, you could do except this testing at the Level a, I've done that a lot doing the show me that jam. It talks about the programmers, done thinks they're done with the story and they asked, pester hey, can I just walk you through what I've done and just a few minutes.
You may have had that. Aha moment of oh I didn't understand that quite writer. Oops I forgot a piece. You've prevented a bug. You haven't even checked the cut into the source code control yet. So having that close collaboration. It's like this iteration testing coding, testing coding.
When we really think the stories ready to go and we get enough stories in a ER now we can start into that quadrant three, the critiquing the product but still business facing exploratory, testing usability, testing things. We really need for code to have been written and deployed. Again more collaboration involving more people. One of my recent jobs, when somebody thought they had a feature that was pretty ready to go, we just opened it up to the
whole engineering organization. So hey, we're going to do another sample. Testing session exploratory testing on this feature for 30 minutes at this time, please join us. We have people rather tame join fresh eyes. I've never seen the future before that's so important. And just as such short time, we
would flush out anything. Any bugs that were there any hidden assumptions that we hadn't thought about getting that collaboration even from people with them teens and then the technology Pages stuff you might do at the beginning. Maybe we've got some new piece of a product that we want it needs to support a large number of users at the same time, it had really good performance. Well we have to think about the architecture and how do we know that?
Architectural scale. So maybe then we Spike the code. We don't do the careful thing. We just do some throw a code with that architecture and now we can do performance testing on it or load testing on it. See if it's scales and if it does we throw away that code and we start running the real code.
If it doesn't we Spike. Another architecture those technology facing tests may be done at the very beginning if that's the most important quality attributes who are business and those become guiding development. So the key thing that I observe when you Lisa is that is still
collaborative, right? It's not just one person or one team who is responsible to come up with all these tests it could be from product manager, it could be from tested, it could be from software developer designer and whatever the role is in the team. And again this the feedback, right? So the test sometimes drives back the development and also by zebras are so, I think I like this concept of holistic starting already.
Thanks for sharing this concept. So you mentioned a couple of times about exploratory testing maybe this term, I think little bit vague for some people. Well, some people think it's like okay we just do any random test. See whether the system crash maybe a little bit of Guiding Light here. What do you mean by exploratory?
Testing. Well first of all I would recommend everybody read Elizabeth Hendrick since book, explore it. I like the fact that she's actually got a chapter in there for programmers, how to explore when they're testing themselves. She has so many good ideas. One of the things that I really liked and I use to guide me is the format to use Explorer. What am I going to explore with chooses a word resources? I think variations. But what is it that I'm exploring with and then to
discover. So when I put a exploratory test Charter together I use that format because it helps guide me. It's a mission statement, it's not necessarily easy to do because you don't want it too small. You don't want it too big. What is it? That I'm going to explore and it might be a specific risk. We think we have some security issues in this area. So, maybe I'm going to be exploring certain kinds of exploits or something, but
maybe, I want to see something. Say, we're adding this form and we got people's names. So, if I use the agile testing Fellowship, our website that we have. One of the things we did was we wanted to make sure that we could use any names. So, for example, your name doesn't have any funny characters but if I put in somebody from Thailand, they might have some different characters or from Sweden and Norway, they have those Different kinds of characters, will it supports every name?
And so we just put an exploratory test Charter to do that. I also like to think about it, as if I took that Charter and I tested, I would have my own notes, I would have my own findings. If I gave it to Lisa, she would explore differently. She would use different examples. She would have her own. So there's enough leeway that we would possibly test differently, but we're still Exploring the same thing and so it just takes practice, but there's a mission,
not just randomly hitting keys. I want to look for this specific thing. Yeah, they're clearly be value and ad hoc testing a lot of teams do bug bashes for they just get a bunch of people and they start hammering on whatever they wanted release that's fine. They'll find back to that way too, but the exploratory testing is designed more to try to find
those unknown unknowns before. They bite you in production and be Organized. I actually got a great tip from a programmer that I work with few years back. When writing the charters, I like to start thinking of what resources I'm at test with first and that helps, you think of that mission. As Janet says, what haven't got my disposal that I can use to test this?
Maybe I've got some API endpoints I could use or maybe I've got some production like test data, I could use or some personas that the marketing department came up with that I could use. So think about the resources, is one help because it's hard to learn, right? Those things and I would also recommend pairing was already on it. I would also put all those Charters as stories in your project, tracking tool in your backlog. Along with your feature stories,
they have to be visible. Everybody needs to know that those have to be done along with all the coding and all the other testing. We need to do that exploring as well. Thanks for clarifying this concept because many people still just assumed exploratory testing. Yeah, okay. You just go ahead and try to break the system or do random stuff. So I think there should be a path. Loud chatter Mission, you keep mentioning that. I think that's a very good
explanation. One more misconception with Janet mentioned is about testing in production. Again, it's got common misconception, people think oh maybe we should not write tests, we just deployed, maybe let people figure it out. Maybe a little bit of Guiding Light here as well, but I remember listening to a keynote years ago at a conference and they said testing is dad, let
the customers test. I wanted that person in the worst way and I kicked myself now for not putting my hand up and saying something, but I was hoping that person would say in my context because that is so important. So testing and production, if I'm doing a Google Maps, yeah, let the customers figured by the bugs and report him, but if I'm doing code for a heart monitor, for example, I'm going to have
way different risk profile. We're going to do all the testing we possibly can before we put it into production. And as such but testing and production and least I can talk about it much better than I can most of the time. But really if you have a safe way to do it, things like feature Flags, right? So the customer actually doesn't see it. We're not touching customer data where we're testing and we have it turned off. So there's different ways to do it.
I think Katrina cookies book testing and devops is still the best one that I've come across for testing feature flags and working within that construct. Yeah, right. It'll guide for testing and devops that's available, van, leave Heaven. You Can Get It Free. She doesn't require you to pay although it would be nice to pay because it's one of my go-to books for sure. But it's so important because as good, a job.
As we do have testing, we know our test environments, don't look like Productions. We're not going to test everything. We're going to miss things, and so we need to be able to put things in a production environment. You some kind of really strategy that. Let's just do that safely. It takes build an infrastructure.
And again, that's an effort that your whole team needs to get involved in. And that's when you Need to have operations specialist site, reliability, Engineers platforms, and ears in your team are working with your team to help you with that. I think you just said something that really struck with me there. Lisa, you said, in the production environment, not necessarily with production data and I think that's the key is the production environment, not the customers data, my last
full-time job that I had. We could not test and production in any way we just could not As if I as a little Services application, there was no way to do it. If we had tried to do it with a got in trouble. And so then it became really important to be able to use our production data observability, what our customers doing power, its does think performing. And also the analytic tools, what are the individual users? We could actually see what they
were doing. Like. Oh, people are complaining about this and we can't reproduce it in our test environment. Let's go see what they're doing in production. Thank goodness. We have those tools now because if Can't test it in production, you need some way to understand what's happening there context. So important things will emphasizing that again context.
So for people who learn about testing production, it doesn't mean that you just deploy your code without any kind of safe mechanism like feature Flags or maybe robust monitoring and observability. Because without knowing what your system does, its kind of like risky, especially if you're dealing with Finance or health and all that. So unfortunately, due to time we have to wrap up.
It's been a really great Conversation about testing in overall but before I let both of you go normally I would ask this question called three technical leadership wisdom. So I will leave it whether you want a combined effort or if you want to go individual. So what will be your tree technical leadership? Wisdom to share with us? Yeah, I thought that was an interesting question and I think my answer is different now, but it would have been a few years
ago. One of the things that we've learned in the last part, 10 years now, but people don't pay enough attention to. It is a prerequisite for success. This fall software development is psychological safety. We had to feel safe to experiment and learn we have to feel safe to ask questions. When you don't feel safe, you're going to burn out. My advice is, if you're in a toxic environment where you don't feel safe, where people are not safe.
Try to be an agent for change. I've really found a good help from the books by Linda rising and Marilyn Mansfield has changed. And we're Fearless changed patterns to help influence people. You can maybe influence people and if it doesn't work, if you can get out, Out. Now I realize that's a privilege. Some people they need to pay their bills and they don't have any other options but also realize it isn't your problem. No amount of yoga and meditation
is going to fix it for you. Don't blame yourself. It's a problem with your organization. So pay attention to that before you burn out and really suffer the anger teams going to be suffering too, that's one of them. I'll let you do one Janet to, of my go right in with that, one of the things was small, f. Can bring about big changes, don't underestimate your scope of influence. The other one that kind of ties in with you two, were talking about psychological safety.
Don't be afraid to try new things, keep learning many years ago and this was in my pretty early days out of high school was my second job out of high school. I was in administer stencil administrative assistant and I had a really lazy boss. Really lazy people kept saying, why do you work for him? What I ended up doing was, I learned because he had me do most of the budget reconciliation and things because he didn't like to do it.
So I had to do it for him but then one day he was sick and it was a very important meeting of all the directors and a budgetary kind of meeting. And they were talking about it. I walked in with all the papers and they said, where's Brian?
And I said he's sick and they said she would cancel a meeting and I said, no, let me explain because I had You can on that extra thing and done it. Learn that it just opened up a whole new world for me. I was viewed very differently than I had been the day before it paid off. So sometimes don't be afraid but then take that jump take that opportunity that is in front of you. Don't be afraid to take it but that's only good if you have
psychological safety. As Lisa said before my second big piece of advice, something I have followed my elf for a change. I think my own advice. The last few jobs I have done, is build relationship. One of the things I got from Katrina cookies, book is about relationships within your team, but also outside of your team. I've gotten so much benefit from asking people on the platform team. Could I have a one-on-one with you for 30 minutes every couple weeks?
So helpful, I learned so much from them. I got so much support from them. That help my team. I just felt like it was a secret weapon. Do start with people who seem friendly and seem like they'll Your allies you got to pick them out the just forms a foundation for so much. The kind of learning that Jim is talking about and helping you get changed, because maybe there's something you want to do, and you can't interest anybody on your own team, maybe somebody on another team wants
to help you really beautiful. So, everything aligned together, I can see the Synergy between both of you, right, as you can tell from the books and all that. So for people who want to follow up with you or maybe continue this conversation outside of this episode, is there a place where they I can find you online. Lots of places, we're both on Twitter. My Twitter ID is Janet Gregory chca for Canada because I'm Canada. People get that confused. Lisa. Your Twitter ID.
It's just Lisa Chris and I'm Lisa Chrisman on all social media platforms that I'm on. We're on LinkedIn for any of our websites. We all have a contact us. My website is Janet Gregory .c a or agile tester, .c a or the agile testing fellow. And I'm Lisa Crispin.com are all connected together. If you go below it's pretty easy to find and Lisa and Janet also has a limp up book so it's the a
gel testing condensed. If you ever want to have this condensed Concept in one goal, I think it's also a good place where you can get all these resources one last question. One fun fact. So when I read your book, I always see this dragon and Donkey. I have my hypothesis, but I'll let you explain what all this about dragon and Donkey. Well, the donkeys are kind of my
mascot because I have donkeys. So we have for donkeys here and our little Farm in Vermont. I've taught me a lot over the years about trust and Agility. I pitched them up to carts and wagons and drive them and take them places and they be work around the farm. There's sort of therapy donkeys. I took him to senior centers and schools and things like that. So yeah, it's just my passion. So three of those donkeys are
miniature donkeys. Yeah. And then she's got one big one who protects the three little ones and for those kind of cool. And the dragon is because I'm a fantasy but file of fantasy stories. My favorite series is a dragonriders of Pern, by Anne McCaffrey. So, I like friendly dragons, not like the Game of Thrones dragons, the friendly ones. So it's just kind of something that appeals to me.
Thanks for explaining that really fun fact, for those of you who also notice in the book that whatever you see dragon and donkeys, this is what it means. Again really Pleasant to have this conversation. Thank you so much for spending your time Janet and Lisa goodbye for now. Thank you so much for having us. It was a fun conversation. It was Thank you for listening to this episode and for staying, right until the end if you highly enjoyed it.
I would appreciate if you share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you are new to the podcast, make sure to subscribe and leave me your valuable review and feedback. It helps me a lot. In order to grow this podcast better.
You can also find the full show notes of this conversation on the episode page at technology Arnold death website, including the full transcript enter, Testing courts and links to the resources mention from the conversation. And lastly, make sure to subscribe to the show's mailing list on pack leader. No dot f to get notified for any future episodes. Stay tuned for the next technology. No episode. And until then goodbye.