Hi there, I'm Richie, your podcast host. I'm a people and tech enthusiast, trackie, and someone who believes that software isn't just supposed to work, it's supposed to matter. This time in my podcast I had the pleasure of speaking with someone very special, Isabel Evans. If you have ever been to a testing conference, read a paper or flip through a book on quality, you've probably come across her name. is one of those rare voices in our industry, sharp, kind and endlessly curious and humorous.
At the testing retreat we talked about stereotypes in IT and testing. Are testers really that quiet, in socially awkward introverts we've always imagined or is that just a myth we've never questioned? What if testers have hobbies like acting, boat building, urban planning or even music composition? Maybe the real issue might not be the testers at all, but our hiring and recruitment filters. And if that's so, what can we do to open our minds there?
And let's look what all of this has to do with testing tools and the illusion of usability we often talk about. This is a conversation about breaking boxes, seeing people and building a more human future in tech. Are you ready of challenging some of your old thinking? Then let's go! Hi Isabel, nice to have you on the show! Well, thank you for having me. I'm really pleased to be here. Yeah, yeah, very nice. It's the first time we met in person here. It is, isn't it? Yeah.
Yeah, yeah, yeah. We are here at the testing retreat, the 28th testing retreat in Belgium. Yeah. Yeah, it was very nice and you gave yesterday a short insight into one of your works now. and you're dealing with how to break stereotypes. Yes, and so please tell us here what's going on in there. All right. So it's part of the research I'm doing. So it's one of the outcomes. I'm not sure where to start here.
One of the things that has happened to me with the research, and I get a feel it might be quite usual, is you have a hypothesis, and as you're collecting data, parts of the hypothesis aren't right, and then you say, "Oh, well, actually, "well, I need to check this thing over here," and collect some more data, and then you find patterns for more hypotheses and test them. You're kind of looping round, iteratively. And so, from the research, which started in 2017 and we're in our 2024.
I started to see various different patterns. I was originally looking at testers and their experiences of tools. One of the things came out of that was that there was perhaps a lack of clarity about actually who was doing testing. So you've got the title, tester. There's actually lots of people doing testing. They may or may not have that job title, but even people who have got that job title, they're not all the same. But I hadn't really got the evidence for that. It was hunch and experience.
So I ran an industry survey and that went out via online networks and conferences and so on. Collected data from open questions, I was asking people about things like how they did their testing and the approaches they had,
and so on and so forth, the tools they were using. But I was also asking in that survey questions about their hobbies, their backgrounds, to find out about them as people, as opposed to how they were doing their work, and then kind of even start lining it up, correlating things or finding that they don't correlate. Out of that, I started seeing that testers were from a very wide set of backgrounds. So not just my hunch that that might be the case, but actually the evidence of it.
People coming in from boat building, theater studies, international relations, urban planning, artists, and so on and so forth. and a really wide range of hobbies. Arts hobbies, very, I'm kind of losing my track here. A very wide range of hobbies and people with multiple hobbies and so on and so forth. You say, well, why is that of importance or of interest? Obviously, we're a wide group of people with wide interests and all the rest of it.
But there's actually a set of stereotypes about who is in IT. Some of those are encapsulated in recruitment and career advice databases. I came across a paper somebody called Machesney had written where they were looking at IT people in general. They were looking at people who wanted to be in IT and people who actually were in IT, and comparing those people with, if you like, the stereotype of an IT person that was contained within recruitment databases and careers advice databases.
And what is the stereotype? Good, thank you. So the stereotype was very much saying that people who are going to be good in IT would be very likely to people who didn't have very much social activity, weren't interested in the arts, weren't particularly communicative, had perhaps very often one hobby or interest that they were quite mono-fixed on. So quite a nerdy, narrow, particular sort of person.
And so when you look at those careers recruitment type databases, he's actually saying, or if you're this sort of person, you should go into IT. Now, when you look to what we want to do, what many people want for IT, there's talk about diversity and inclusion and talk about, if we're representing the world at large, humanity at large, if you've just got one narrow group of people doing the work, is that sensible? So, McChesney had done that work to look at that.
What they found was, a relatively small percentage of the people in their sample who were IT people met that stereotypes. There were a wider group of people working in IT. I'm having an awful old age blank brain moment. I think it was, let's say it was 30 percent. So, maybe 26, 30 percent, something like that. Let's talk about it being That kind of amount.
So, you know, a significant proportion met the stereotype, but it wasn't everybody, it wasn't the majority, by any means, nowhere near the majority. And when you look to the people who are aspirant to be in IT, they were closer into the stereotype. In other words, the recruitment process was narrowing down the people who were gonna be encouraged to come into IT or felt they had a right to be in IT. So I took the modeling that Machesni had done with their data, and I reapplied it in to my data.
So I wasn't expecting to be doing this, but it's like, "Oh, wow. Okay, let's try this." When I did it with my sample of testers, it was only six percent of them that met that stereotype. In fact, when you looked at things like the number of people who had arts-related hobbies. It was enormous. Huge number of people have arts-related hobbies. Even within that, the other thing about the IT stereotype was very much about being quite passive in those interests.
I think that doesn't fit with what I know. As you look at it, so you take something like a love of music as part of those interests. There were a proportion of people amongst the testers who were talking about listening to music. That's quite passive, but it's an arts interest. There were another group of people who were more active. So they were playing music, they were singing in choirs. That was quite interesting, quite a proportion of those.
But then you'd also got people going, "Oh, I compose music, I write songs." So they're actually really actively engaged and creative. And that was true for a number of those different interests. So that was fascinating to me that people just weren't, from that point of view, meeting the stereotype.
And then when you actually looked at people's backgrounds in terms of what degree they'd done, there were people with arts degrees, with science degrees, with social science degrees, with arts degree-- sorry, with IT degrees, and also people without degrees at all. So a wide range of backgrounds and people who'd done a wide range of jobs beforehand.
So, yes, the original version of the paper was called "From Artists to Urban Planners," and then we actually changed the title later to "Breaking Stereotypes." When you looked at the roles that people were taking, I mean, I guess you might think, "Oh, if they're coming in as arts graduates, they're probably going to be doing more sort of test management, maybe some test design, but not so much of the technical roles."
Forty percent of those arts graduates were actually doing test automation, technical testing roles. When you looked at the IT graduates, around about 40 percent of them were not doing technical roles at all. So it's like the degree doesn't predict what you're going to end up doing, or your aptitude for doing it, or your enjoyment in doing it.
But one of the things that was also interesting, and this was like it was another bit, while I've got this data, I might just try modeling this and seeing, can I find any patterns? So I was looking at communication styles, and there was a bit of a correlation between people's degree subject and how they express themselves when they were talking about test approaches.
So the arts graduates wrote very easy to read, well-expressed, almost I'm going to say essay-like, but storytelling descriptions of what they were doing, and they talked a lot about people and the problems they were solving. The social scientists very much talked about the organization as a whole, and teams, and how people were interacting. The science graduates, they made ordered lists, like this is what I do in this order, as there's a real structure to it.
So it's quite terse but very informative. The people who didn't have degrees were very much into storytelling. Almost like, "I've got a glass of beer and I'm sitting here, I'm telling you all about it. I'm going to be writing it, but that's what's happening." Now, the IT graduates, loads of technical detail, not very well-structured, quite hard to get through, and not very storytelling.
I thought, "Wow, actually, We need those arts graduates and those social scientists to get us communicating across the organization and telling our story, and we need those scientists to give us that order and structure. The IT graduates are bringing in the technical skills that they've learned, but maybe not those other communication skills.
I'm a computer science graduate, and I know I'm chatting away now, but actually in my kind of natural state personality, I'm much more over in that solitary kind of Sheldon Cooper side of the world, you know? But I thought, "God, we need those other graduates. "We need those people from the other backgrounds "in order to be able to put across, "these are the risks we found, "here's what we need to communicate to you.
"What do you need from us?" - And it leads to a very different and diverse perspectives. - Yeah, absolutely. and perspectives that reflect the rest of the world. - Yeah, yeah, yeah. And you put it in, it was the base was from your work about the test tools. - Yes. - What were the insights from this diversity from the tester to your work? - One of the earlier findings that led to that survey was a realization that first of all, testers lived experience of using those tools.
So their user experiences, the users of the tools knocked onto a poor lived experience. So it's like it was damaging. There's a lot of frustration, a lot of upset. There were some good emotions as well, but it was very high emotional response. Also a set of findings. So it's important to do something about this for people's health and welfare at work. But also that I thought usability was going to be the problem, and it kind of was, but not in the way I expected.
So one of the things was the highest scoring, quality attributes of the tools that people talked about was the operability. In other words, can I carry on my workflow? Can I carry out the tasks to reach the goal that I have as a human being with this tool. That was the most problematic and the most raised. You infer that from the way people are talking about it. You pull it out from what they've reported. It's not that you've asked them is operability a problem.
You've said, "Describe to me an experience with the tool," and then from that you pull out what actually the technical and quality attributes they're talking about. So it's not directly ask questions. It was all open questions. Also, one of the things that was really causing people the most annoyance was tools that had been bought because they had really pretty interfaces. So superficially, the usability is there, but when you start using it, it's not supporting how you do your testing.
So out of that came this concept of the illusion of usability. So I started talking with my academic supervisors about how do we deal with this? And we started wondering whether a UX or academically an HCI perspective, putting that lens on test tools would help to understand it. And when you do that, you start asking who is this tool for? What are the characteristics? What are the persona groups that we're dealing with?
and as I started investigating that, I initially thought, "Right, I can set up a framework that's going to help tools designers run this information through and it'll come out and say this is." It's too hard. There's too many variables. I can't do it. If you can't come up with the ideal test tool or even like the framework for helping you design the ideal test tool. Then I had a meeting with somebody who's an academic at another university, a guy called Hossein Dugan.
He was listening to what I was doing in the work and he said to me, "You know the contribution could just be a set of heuristics that people consider when they're designing a test tool. Don't try and write a whole framework. Don't try and write a tool for writing tools. Don't try and solve the problem. Go away and come up with what are the heuristics based on the evidence you've got."
So taking all the data, I've come up with 12 heuristics, which are couched as questions because they haven't got a single set answer. It's actually around, you need to ask yourself these 12 questions, and then where that takes you, will tell you how you need to design the tool. Those are now out of a repository. I'm going through industry case studies with people using them.
They've been through several expert reviews and iterations of refinement, and they'll go through a final set of expert reviews, and then be published out. My vision for them is that they're out under some sort of Creative Commons license. So everybody in the industry who is designing tools in-house, who is designing open source-based tools or vendors, or people who are evaluating tools can pick those heuristics up and use them in any way they want.
Yeah. To help them think about how am I designing this tool or how am I evaluating this tool. That's going to be the final product out of the research. Yeah. It's just going to be there available for everybody. Also, I've realized in designing them, It's not like you go through 1 to 12 and you've got the answers. In themselves, they're different.
I've seen on the different case studies, people using them in different contexts, picking out particular ones as being important in this context, picking out a different order to run them in, revisiting them later in the evaluation process or the design process because they realize they've got to go back and revisit a question. So I hope really flexible as well, but a tool if you like, the set of heuristics is a tool which people have to use thinking.
They're an aid to thinking, they're not a set of answers. So fingers crossed with that, hoping to get all of that done over between now and the end of the year and then really be pushing it out, writing up for my thesis and then publicizing them more during next year. - Oh, great, great work. I like it. I read it last day, yesterday, through the questions. It was very, very great. And I think also very practical to use it.
So it's not very, not so high level, but you can use it in a pragmatic, practical way. - I hope so. I mean, if I can ask you, I mean, you said you looked at them. I mean, for you, was there a particular question in there where you thought, "Oh, actually, I haven't thought of that before." Or was it confirming or was it more, were there things which you thought, "Yeah, that sparks something new for me." >> I think it's written down. When I think about tool design, it's very intuitive.
This is now written down the questions and give good orientation. Right. Okay. Okay. That's interesting. One question back to the stereotypes. You said that HR always in recruiting, doing very, very fixed on the stereotypes of IT people. So what do you recommend? How can we soften that up? How can we transform it? Because we need diverse testers and IT people. I think one of the things is to talk to HR people and talk to recruiters. And I'm hoping that
the paper can give people evidence that they can take. And not just for recruiting testers, but actually recruiting developers and UX people and whoever else you need in the product owners, systems analysts, architects, and so on. So I hope the paper gives people things where they can pick it up and go, "Right, here's a piece of evidence. I can take that." I'm also at suggestion of the audience when I gave this talk at the HUSDEF pre-conference meetup a couple of weeks ago.
what the audience wanted me to do was take it to HR people. So I'm actually just in conversation now with some HR people I know, and I'll talk to them and say, well, where would be a good place to take this information and how to show it? But again, the paper's out there. So people can pick that up and take it to their own HR department, their own recruitment people, their managers, and start the conversation, I think.
- Yeah, I think it's a very important part to get future proof IT departments and software developer and testing. Yeah, Isabel, thank you very much for joining the show and tell about your work here. I think it's very, very interesting and it opens up the stereotypes we have sometimes in our head and with your survey, you give now a fact-based evaluation of this. So thank you very much. Oh, well, my pleasure. Thank you so much for taking the time to talk to me because
I'm really delighted because I want to get the word out there. So, thank you so much for taking time to talk with me. Thank you too. Thank you. Then have a good time here. Will do. And you. Thanks. Bye now.