Welcome to Software Testing Unleashed, the podcast for testers, developers and software makers who live quality as an attitude. Get fresh ideas and sharp insights to grow your mindset, to learn new methods and to drive real change in how we build software. software and better teams for a better world. Hi, I'm Richie, software quality coach, keynote speaker and author. My guest today is Maciej Wyrodek. Maciej is a quality consultant, mentor and trainer with a
strong focus on test automation and process improvement. He's worked across a a wide range of companies from startups to big corporations and from in-house development to software houses. That gives him a broad and practical view on software quality. He's also a passionate knowledge sharer, writing articles, speaking at conferences and he has his own YouTube channel, IT Morning. For much learning and sharing are at the core of our work. We first met at the
testing retreat and I took a lot of inspiration from his thoughts. In this episode we talk about switching from Cypress to Playwright and what it really takes to change automation frameworks in a live fast-moving product environment. What do you do when your automation tool starts working against you and not with you? Why is the most popular tool not always the right for you and your context? And can AI actually migrate your test suite or does it just look like it?
We go step by step through March's team strategy from evaluating tools in a hands-on hackathon to managing real-world migration challenges with limited capacity and high expectations. You also hear how Playwright wins out where it still surprises them and how the team experiments with AI to accelerate the transition. And now enjoy the episode. Hi Maciej, nice to have you on the show. Hi Rikki, nice to see you. We haven't seen you in
a few months now. Yes, a lot happened in this month as I heard up front. And we met at the testing retreat in Belgium last year, the second time we met there at the retreat. And we had the idea to make a podcast episode together. And yeah, I'm very happy that you are here and we've chosen a very nice topic for our talk today about test automation and your migration, your big migration
you're dealing with. But before we start, just get us into what's your daily business, what you're dealing with in your software development? - Okay, so starting, I am in software development for about 14 years, if I'm counting correctly right now. And currently I'm working for Displate. Displate is a company that is making metal posters, as you can see behind me there, those are the posters from Displate. And you could think that, Well, this is basically e-commerce shop.
There is not much development there, but actually there is a lot connected to, you know, since we have our own factory, there is a lot of our custom admin staff, like, you know, managing artworks, managing the copyrights, managing work with the artist. So there is a lot of custom code there. So it's not just, you know, simple e-commerce. And for part of this, as you mentioned, we had the end-to-end automatization in Cypress, and we decided to do switch on the playwright.
I can go for why we decided to switch. You can go through the whole process if you like. Yes, I would be very interested, because Cypress is a very often used tool for test automation. Why did you choose to go away? And what is your amount of tests you have in Cypress?
- Sure. So basically, I need to start a little before Mike came to the display, because at that time there was in one of the projects, let's say the front-end part, the part that most users are using, there were some automation in Cypress around the time that I joined the company, and there was another project, the backend stuff, not backend in the understanding of us as, you in understanding of the business. So, you know, the place where the customer support and others are working.
And that project hadn't at that moment, this automatization. And one of the first things I was there was deciding how we're gonna automate it. And what I'm now calling bad decision was I decided to go with the Cypress back then because I was new in the company. It was kind of my decision because I joined in as QA lead, but I didn't want to rock the boat and take another framework on one thing we have in the Cypress, another in Playwright.
From the hindsight, it was a kind of bad decision due to the nature of the project, because Cypress has its lots of advantages. For example, it's very fast to learn, but it has very specific DSL, domain-specific language, that you have to use. It's very hard to write test in different way then the Cypress authors want you to write. They have very opinionated way on writing tests, how the assertions should be looking, what tests should be testing.
And basically, lots of people are going away from that because there are thousand plugins and you can do everything with Cypress. But as with the plugins, you know, the problem is keeping them up to date, keeping them comparable with the different version of Cypress. And what we were seeing is that our way of writing test, what we wanted from our test, what we kind of needed was different than what Cypress was able to offer us.
And quite often we were going not with the framework, but against the framework. So for example, since we have some third party like payments, which often are iFrames, Cypress doesn't like iFrames. You can work with them with proper plugin, but they are always problematic. There were also some aspects that Cypress, for example, what we needed due to the size of our web page. We, for example, needed a whole page screen.
When something failed, usually you wouldn't have it on the screen because it was checking element that was under the screen. And the way how scrolling works in the Cypress, you won't always have the proper position of the screen on your errors, because I don't want to go into technical details too much because probably some of your fans know more than me.
But basically, Cypress is properly scrolling to that place, but it doesn't mean that on the, let's say, error view, you will have the scroll DOM. And that was a big problem. And if you want to set up the proper way of scrolling the page, you can set up something like full screen error. It doesn't always work. There is, maybe now it's fixed. I haven't checked it in some time because we're doing the work around a year ago.
There was a known issue, which back then was kind of won't fix that if you're using React with specific configuration, if you try to do full screen page, like, you know, the whole page that is very long, it will be repeating always the first view. So you will have, let's say seven times the header and just it. And well, for one, it's problem for us. It was problem for others. It's something that they can live with.
We had some workarounds around it, but we had their own workaround on workaround, workaround. And the point that kind of was for us throw that broke the camel back was the situation with the currents. I don't know how much you know about how the Cypress parallelization work. No, tell me, please. - So basically, in Selenium times, you basically were completely on, you could do everything. You had the Selenium grid, which you could self-host and you could do your own parallelizations.
There were companies and they're still like BrowserStack, Sourcelab, that they were providing your Selenium grid options. And Cypress kind of decided to make business on selling parallelization. So framework by its own, it's free to use. You can create your own plugin to do local parallelization, but kinda, and now it's much more popular. Back then there wasn't many of such options.
One of them was Sotus/Cypress done by the, I don't remember his name and he's gonna be mad at me because we're talking quite a lot. You know, this is typical blanking during the interview. So, Andrew, Andrew, sorry, Andrew, that I couldn't remember your name. So, Andrew that don't, sorry, Cypress also had the currents. He's implementation that you can buy and you can use for, you know, with their hosting, that's kind of stuff. Kind of similar to the Cypress Cloud.
Difference is that in the amount of users that you would need for Cypress Cloud for organization like us. In Cypress Cloud, it was 10 to 20 times much more expensive than using the currents. We wanted to discuss it with Cypress because basically at that time they had packages that was like you have the number of users and number of runs. We wanted more users. We didn't have problem with number of runs, but we wanted more users.
And they wanted from us to go to the much bigger packet, which has much more runs, which you couldn't use, but they weren't very flexible with negotiating that kind of situation. And that was more than basically my whole QA budget for that year. So we went with currents and at some point, Cypress was very mad at currents, calling them bad, bad, not enemies, bad competition. Because let's not go into the technical details. I am on both sides.
I understand the point of view of the Cypress because as much as it was funny for me, the fact that the library that was used by Karens was called Cypress Cloud was very on the nose and kind of could infringe on the naming of Cypress. But there was a lot of bad blood there to the point that Cypress blocked the ability to use currents. If you have the Cypress above, I think version 13, you cannot use it with currents. And for us going with Cypress wasn't the option with the Cypress Cloud.
It was too expensive. And we had lots of other smaller issues and we were seeing lots of the problems This kind of move showed us that we couldn't treat Cypress as, you know, a serious partner in any form of negotiation. So we bought, we still, we're currently still with the currents on our Cypress that is not updated because we are, I think, on version 12 something, the last that is supported by currents. But at that point, we decided we will be leaving the Cypress ecosystem.
That was, just to show you how long it took, that was September 2023. Oh, okay. Yeah. So, just to show you that the process wasn't fast, it was quite long. Because, for first thing, most of my team is doing typical day-to-day business work, testing stuff that is regular, still maintaining our regular test. So even if that was one of our goals for 2024, it wasn't as easy. It wasn't something that we knew that's going to happen from day to day. So, yeah. So you have no background.
Do you have any question to the background? No, that's a good introduction to the topic. And if you decide such a decision to change the test automation framework from one big player to another, is you have to deal with a lot of things, you have to mention all the transition stuff, and as you said, you have all your day-to-day business every day. So what was your plan or how did you do the transition there to your new framework? So transition is actually happening,
because first we had to say one thing, selecting the framework. Because at some point we kind of all expected that Playwright may be the framework that will win. But actually we... I'm not sure if I'm here. Oh yeah, you're connected because I had something that like stopped recording. But I think I'm back. So again, Playwright seemed as possible the best candidate, but actually we didn't know that. We were suspecting it, but you know how it
is with that thing. So first thing what happened, we had to look what are we using the Cypress, what are the things that we like about Cypress, what are the things that are not working for us. And then from that we went to more generic. What do we want from our end-to-end framework? what is our approach to testing, to create our list of, maybe not goals, but the aspects that are important to us.
And the next step was looking. There was around, I think, at that point, because it may seem like a small part of time, but we had the first iteration of our goals done in the first quarter. Then there were quite a lot of other projects happening. So the second quarter of last year, we couldn't do much with that project. And around, I think it was July, we started to looking for, you know, the tools. We're looking at different tools. At that moment, it was mostly statical.
So we had calls with some companies looking at the tools that are on the market. And I'm going to say I was kind of downhearted. I spent my lots of my career doing research on tools. And when I was in, I can use my company names on your show or I should avoid them. Please avoid it.
Okay. So when I was working at a company that is no longer on the name that I was working there, Basically, I was a Guild Leader of this Automation Guild, kind of in the Spotify model, where the Guild is more like a community of practice. So, and one of my jobs was actually doing research, what tools are there and what are available. And back then I was spending a lot of time on this. And that was 2019. And when we started doing it in 2024, It was like nothing has changed. There is nothing new.
Kind of last big thing that came to the market was play right, which was at the beginning of pandemic and lots of things that were at that time looking like promising thing a third diet or change in the competition, think as. And that was kind of that kind of broke my heart because I was expecting to see such much progress.
If you asked me if I was doing a set of predictions five years ago, I was expecting that lots of tools that are record and play right now, they will be the big part of our market. That most small companies or even big companies for the fast development, they won't be doing coded development, they will be using record and play tools. But there are tools like that. Yes, there is Test Team, there is Map, there is Polish Backpack.io, but they all are very limited.
And you have to have very specific needs to be able to use that kind of tools in your process. So we kind of spent some time on the research. We had big table with different aspects we were checking, which spend what. We were giving them points. Some criteria were much more point-worthy, others less. And we have to select something like, I think, seven tools at the end that we decided to go into, you know, actual hackathon.
Actually, you know, not just looking statically what they have on the page, what are their demos, but actually doing proof of concept on our site. And I think that was, I don't have the timeline next to me, so I cannot just check. But I think around September, we, as the quality engineers, with help of interested parties, few backend developers, frontend developers, decided to finally go, to finally we had time to do the actual experimentation of the tool.
And as you can expect, Playwright basically left everything else in the dust. I'm not gonna use the names of the tools. But one of the coded tools that I was actually, because we had more people than tools, and some tools have few people doing them. One tool was done by me and some backend developer that he is, well, he is backend developer, but he knows frontend other
stuff. And we both couldn't set up one test. We actually spent whole day and we couldn't make a working test, going with the documentation and a quick setup guide.
It was later we found out that they had some big update of the framework, something like a few weeks before, but the documentation was so out of date that even trying to, you know, we actually had to go to the issues on the GitHub repository to do that kind of And that was also already a big no-no for us, because if you have to spend your time during your daily development of the test on basically not finding information on documentation, but actually had to dig through the support tickets,
it's something not right. But I think the most interesting case for me, I kind of mentioned it to you, I think, in November when we were on the retreat, One of the developers got tool that now had some AI component for creating test. And he basically in the language a la BDD had to describe what the test was supposed to do. So something like go to the homepage, select this category, no, select the first product you see on the page, add product to cart, buy it.
Basically, because what I forgot to mention, we had a list of, I think, 10 tests prepared to see how many tests each of us will be able to write and to have baseline to compare because we don't want everybody to write completely different tests. So we kind of had this one. And he's showing us, he's calling us in the middle of day because he's saying, "Either I'm too stupid or I don't understand what's happening." He show us the script.
He wrote basically, he did copy paste of what we had prepared in the words for that system to write. And that tool decided to scan our page. So we are seeing the recording how it's actually automating it. It went to the homepage, then it went to not the product. You had thousands of our products on the homepage because we have the presentation of different plates that you can buy.
It went to some of our menus, went to some collection, then went to another collection, then to another collection, Then to some brand page. One interesting thing what happened on the way, it managed to deal with the pop-up because some pop-up about newsletter or something else opened and it automatically closed it, going further into the site. And finally, after six minutes of clicking, it went on the product page, it added it and it finally added it to cart. - Yeah, so it worked.
- Looking at the scenario, you are like, There is nothing like this on this tool. I later talked to someone that actually is using this tool, not the AI part, but the base tool. And he told me that this is something connected how they're using something, some AI agent like, you know, chat GPT. And if probably you would try it to do after hour, it would generate that test completely differently because what is doing is first using apparently those AI to create the path.
And if you accept that path, it writes it as, I don't remember, I think it was the puppeteer script under the hood, but don't quote me on that. And probably if, until we approve that script, if after our would try, the large language model that is used for the generation of the test probably would generate completely different path for that. So it sounds very, very interesting hackathon you had on this stage. Yeah, I'm actually one thing that I wish we had done. We would record that because
I see it in my own eyes. I even tried to reproduce it later using that tool. I had different stories connected to that part, but I couldn't reproduce it in such a funny way as it was doing it back then.
Yeah, yeah. So it's very interesting to hear that you did such a day with all your QA guys and development guys to make, to choose from these tools for test automation framework, because the choosing a tool is always a hard thing and to do it with real, your test cases with 10 scenarios and try it in different tools on one day is, I think, a very, very smart way to check if the the tool fits your requirements. - But you know, what is the worst part about it?
Then explain to your boss why after doing all this research you choose still the tool that is the most popular tool and was the obvious choice from the beginning. - Yeah, but then you have the proof, yeah? - Yes, yes. And of course, to make it even fun, we did lots of research. We actually checked lots of different things about Playwright and I'm going to tell also the story about how Playwright performed at that workshop, because it kind of went a very good pairing.
Our developer that is currently our expert on using AI to write code, he's even doing internal trainings for us. And he was playing with the cursor and using the also Playwright-- just now I'm blanking on the name. I think it's called Inspector for doing the record and play to record the test and is generating a code. He actually managed to write all the 10 cases and stabilize them to be actually stable during the hackathon. - Okay, great, yeah.
- For comparison, only person that was able to also do the 10 tests was someone that was doing strict record and play to. So actually it was far away the best tool and the best results. And it was, I think in hands of different person, it wouldn't went so well because you know, this is the kind of problem with the such small sample size, but it also was showing the potential of the playwright.
Of course, what is worth mentioning it, those tests generated back then wasn't what you would call might enable. So if something changed, probably you would have to basically record the whole test again, similarly how you do it with record on play, because the code wasn't done as a very meta-enabled. It was basically done to show it's possible. - Mm-hmm, that sounds good.
Did all the guys who did this challenge with you, this hackathon, follow this decision with Playwright, or was there a hard discussion because someone wanted his tool? Actually, there were a few tools that were performing quite well, but at the end of the day we were doing a demo, that basically everybody was showing what they were using, what is their opinion.
After seeing the Playwright in action, it was quite clear that there are a few good options, but nothing that will compete with Playwright. Yeah, I understand. So you decided to go to Playwright, and now you have your big repository on old test cases in Cypress. And now, what was your strategy to go to the new framework? The actual strategy is, I need to go a little with our approach to end-to-end tests.
We have something that is called, we have our full regression that is running, not nightly, it's running a few times a day, basically. running at the start of the day, at the middle of the day, and basically at the evening, because we kind of want to get some good information what is happening during changes done by developers. Because full regression has a lot of tests that if something will be broken for a few hours, in our case, those tests are not like something that will kill us.
It's good for fixing them as soon as possible, but it's not critical. We have the smoked suit that is running after every deployment. And this suit is something that actually, if it finds bugs, usually it's something that means either you are fixing it right now or it should be rolled back. So actually the properly critical thing. And we have something that we call top 10. Those are the 10 most critical paths in our system. Those are the money makers.
The thing is that something is broken there, basically drop everything and go fix it. And those are running regularly on our production environment. Something like during the high season, like Black Friday, so when there is lots of money to be made, they are running every 10 minutes. In outside of high season, they are running every half hour. And we decided to start with those because those are the tests that need to be the fastest.
Those are usually not the easiest test to write, but they are also kind of vertical slice of our system. So, the first thing what is currently happening, because we haven't finished it, the goal was to finish it until the end of December, but due to a few problems, we didn't manage it, was actually the movement of those tests to the playwright. The plan was it's kind of evolving, but move those tests. Set up is it as the main test that are running every 10 minutes.
And then as the new functionality will be coming, we'll be writing the test just in Playwright. And in the meantime, we'll be slowly moving all the tests. So if something will be broken in Cypress, if it's a small thing like change of locator, you will be fixing it in the side press. But if it will be, you know, like change of logic, the proper way is to move the test to play right.
But we are still not at the stage because the top 10 was, I think the last test we are moving finally this week, the week of the recording, not the week of the premiere of the episode. So, as you mentioned, we started moving it. But of course, even after doing all of the research, they were coming a lot of surprises to us. First thing that actually caught us quite off guard was the way how the session is managed in Cypress versus the Playwright.
Because in our case, we have a lot of A/B tests on our site. And we have a lot of additional thing that we are keeping in local storage, session storage, and cache and the cookies. So there is a lot of things that we need to set up for our test to work. Because again, we are doing lots of testing on the production, not only the test environments, but also on production.
I am big fan of the saying that I don't remember who told me that, but there is no second testing environment such close to production like production itself. (laughing) - Yeah. - Yeah, so basically if something can be tested on production and it's not risk to break something for thousands of users, I'm usually fine of testing it on production.
So since that, we had to set up our test to actually first cut off the analytical cookies because if we don't want to do that, this means that our test will be actually collected data. It was even a funny story. I think that I don't have that plate behind me. There was one plate for semi-popular author on our site that one of our testers very liked. So, it was used in a lot of tests that we're doing. So, you could imagine that daily our automation
could go on that page, I think maybe hundreds, if not thousands of times. And being in the In the kitchen one time I'm hearing that maybe they should go to that guy, talk about some promo because a lot of people are visiting his pages and not buying it. Okay. Because, you know, in stats we were showing that we were going to that page. Yeah, great.
So, again, this is old story when we were still figuring out where we should be informing, what information should be in the test to, you know, to not affect our sales data and that kind of stuff. But now we have this information. We have this all figured out in Cypress. Now it's time to figure out in Playwright.
It starts to be fun because at some point, nobody realized that, wait, but Cypress has very good ways of doing the interception of the API request, because we are doing lots of things with the API request for set up and copy cookie and that kind of stuff. Cypress playwright also has the options for that.
But actually I think the way how it was done in Cypress was basically much better because in Cypress, we had, I think maybe two liner for, you know, intercepting the cookie, not the cookie, the request with the, let's say authorization. It wasn't authorization, but let's say for sure it was authorization.
And in case of the playwright, actually we had to spend some time looking at the different documentation, the different suggestions, how to properly set up the way how we're handling the setup of both the requests, how to properly set up the, oh yeah, block hosts. Because in Cypress, it's easy. You have basically in the config option block host and all those hosts are blocked. In a playwright, as far as we know, there is option that we basically missed something. Doesn't have such functionality.
We had to write it on our own. So we had to find how to properly intercept the request and how to do it in the way that it would be always the working there. So we have some mechanics for that, but it was like, it wasn't even in our spreadsheet of things that the framework should have. Because we assumed that if we have, it can handle the requests, it will be able to handle that kind of thing. So it wasn't a separate position in this.
And that was kind of thing that we are still visiting because you have one thing like that, then you coming another thing. And then something that actually was for me very funny, old tests, because lots of tests were written a long ago. I think one of the tests that we're moving, it's older than I was in the company. And at start, we had that decision made, but it was forgotten. We tried to rewrite the test, so we wasn't doing it one to one.
So we are looking at the logic, but we shouldn't be taking much of the code. But at some point it was lost. And people were trying to, you know, kind of rewrite Cypress code in play, right? So it's like trying to put, you know, the square back into circles. So this is, I'm not gonna say that it's kind of fault of anyone. This is, you know, just basically standard work in the pressure, the deadline is coming, you're starting to, you know, to think differently about stuff.
You still are not fully trained in the new tool because we did some training, but you know how it is. You still are learning how to use. So a lot of that kind of stuff is coming. The way how it's called, the data management is done, a lot of that kind of stuff that you think, all frameworks are doing it kind of the same. No, they are not doing it the same. (laughing) - Yeah, I understand. Yeah, so you're dealing with some surprises on your transition way. But what is your strategy?
So if you, I think before you started the transition, all your test automation engineers had a lot of work to do anywhere the whole day. So how do you deal with creating new test cases and make a transition of the old ones? How do you deal with that? Actually, we were kind of happy. The reason why we decided to use most of December for moving the tests, for doing the setup of the top 10 is that it's holiday season. For us, it means that there wasn't as much feature development in component.
There was a lot of cleaning, a lot of doing that kind of stuff because as with every e-commerce, kind of moment from a few weeks before the Black Friday to kind of Christmas is the moment that the companies are doing most of money. So you don't want to put on to put any big feature outside because you don't want to lose that time window of selling for possible
issues. So that was one of the thing. Another is basically prioritization. We are a smart team, QA-wise, because most of the developers are doing their own testing. We don't have a tester per team. We are kind of working in a similar way to DevOps team, that we are a platform team that is giving support.
We're doing research into, for example, how to test payments in different markets, because let's say we are going to start using some payment method in America, and the team has no idea how to test that. So usually they come to us and A, figure out how we can test that. Or they come to us A, the situation is like that. We have no idea how to make sure this will be tested. And that's why we kind of have more, we have the time to do that kind of maintenance stuff. But it depends.
I'm already seeing the roadmap for the next few months. I would expect that probably in February, March, there won't be much time on the automation. Right now, we're kind of doing rotation, that two people are supporting teams with their issue. One is writing automation and we're doing rotation that everybody can do some automation tasks. But I don't think it will be possible for the February and March.
Right now, I'm kind of trying to, you know, still working at the team capacity, have some for that task. But we will be seeing how it actually will be working. So my strategy is kind of looking at the backlog and having some capacity frozen for that. Yes, yes, that's great. And if you look at the upcoming year 2025, do you think you will manage the whole transition this year? or is it not possible or is it just not relevant for you because it goes like it goes in the flow?
- Actually, that is very hard question. Personally, I would hope we will manage to move most tests in 2025 to play that because what actually was supposed to happen last year but didn't because we knew we will be leaving Cypress is get more front-end developer into test automation. It's actually happening with Playwright right now. Developers are participating
in the pull requests. Until we finish, you know, stabilization, the top 10, we don't want them to write tests yet, but we will be planning to give them ability to write, maintain their own tests. So that will be also fascinating for us. But-- and this is giving me hope. The other thing that is giving me hope is an experiment done also by that developer I mentioned before.
He did an experiment that it's still not working properly, but with some maybe a few more tries it will be using the cursor, which is the IDE with built-in AI for translating tests from Cypress to Playwright. And he did a few examples on his own. Those were great, but when he wanted to do the demo for us, it didn't work. It actually on the demo, as you know, didn't work. But I'm seeing the potential there. Maybe it's either we need to create better prompts or something else.
but basically what he did, he took our Cypress repo, do copy paste inside the Playwright repo, and then he told to discuss it with the AI agent. So this is the Cypress repo, this is documentation, here is the part that is Playwright. And now I want you to help me, the test is for example, homepage smoke test or something like that. And I want you to create for me the version of the test in Playwright. And after a few tries, it managed to do something that actually worked.
My problem with it is I'm still not convinced about maintainability of the test, but I'm seeing it more as we learn probably more the ability of cursor as the whole team. I'm expecting that the speed of writing those tests will go faster. Yeah, great. So maybe we can make a future episode about this, about the experience with your final migration and transition to Playwright with AI support.
- If you want, I have one interesting tidbit because I was playing with the cloud desktop, the cloud, you know, the AI agent, they did the desktop agent that you can control your desktop and I was doing some experimentation how to use it to, you know, for test automation. Okay, so maybe we have a future topic here. Yeah, Maciej, thank you very much for all these insights. I think it was a very, very practical hands-on episode.
I think for a lot of test automation engineers out there to get an idea of what to think about the hackathon and the transition from one tool to another and your thoughts about why you're choosing which tool and why we're leaving that was very interesting for me too. And yes, I wish you the best for the rest of your transition and for this for this whole year. And we will see in person in October or November, I think again. October, I think it's October.
It's October. Okay, so a little bit up there. And yeah, I'm happy to see you then again. And thank you for joining the show. Thank you. Bye. Bye.