Hello and welcome to a new episode of Podcast Software Testing. This time we let some international flair flow in again, because I have Mace Virodec as a guest today. I know him from the Testing Retreat 2024 and he talks a little bit about his everyday life. Namely how he and his team have made the migration from the test automation framework Cypress to Playwright. What considerations were there, why they were moved at all and how the whole thing works. And now have fun with the episode.
Hi Mace, nice to have you on the show. Hi Rikki, nice to see you. We haven't seen you in a few months now. Yes, a lot happened in this month as I heard up front. We met at the Testing Retreat in Belgium last year. The second time we met there at the retreat and we had the idea to make a podcast episode together. And yeah, I'm very happy that you are here and we've chosen a very nice topic for our talk today. About test automation and your migration, your big migration you're dealing with.
But before we start, just get us into what's your daily business, what are you dealing with in your software development? Okay, so starting I am in software development for about 14 years if I'm counting correctly right now. And currently I'm working for Displate. Displate is a company that is making metal posters. As you can see behind me there, those are the posters from Displate. And you could think that well, this is basically an e-commerce shop, there is not much development there.
But actually there is a lot connected to, you know, since we have our own factory, there is a lot of our custom admin staff, like managing artworks, managing the copyright, managing work with the artist. So there is a lot of custom code there. So it's not just simple e-commerce. But for part of this, as you mentioned, we had the end-to-end automatization in Cypress. And we decided to do switch on the playwright. I can go for why we decided to switch.
You can go through the whole process if you like. Yes, I would be very interested because Cypress is a very often used tool for test automation. Why did you choose to go away and what is your amount of tests you have in Cypress? Sure. So basically I need to start with this before Mike came to Displate.
Because at that time there was in one of the projects, let's say in the front-end part, the part that most users are using, there was some automation in Cypress around the time that I joined the company. And there was another project, the backend stuff. Not backend in the understanding of us as developers, but the servers. But backend in understanding of the business. So the place where the customer support and others are working. And that project hadn't at that moment test automatization.
And one of the first things I was there was deciding how we're going to automate it. And what I'm now calling bad decision was I decided to go with the Cypress back then. Because I was new in the company. It was kind of my decision because I joined in as QA lead. But I didn't want to roak the boat and take another framework on one thing we have in Cypress, another in Playwright. From the hindsight, it was kind of a bad decision due to the nature of the project.
Because Cypress has lots of advantages. For example, it's very fast to learn. But it has very specific DSL, domain-specific language, that you have to use. And it's very hard to write tests in a different way than the Cypress authors want you to write. They have a very opinionated way on writing tests, how the assertions should be looking, what tests should be testing. And basically, a lot of people are going away from that because there are thousands of plugins.
And you can do everything with Cypress. But as with the plugins, you know, the problem is keeping them up to date. Keeping them comparable with the different versions of Cypress. And what we were seeing is that our way of writing tests, what we wanted from our tests, what we kind of needed, was different than what Cypress was able to offer us. And quite often, we were going not with the framework, but against the framework.
So, for example, since we have some third-party, like payments, which often are iFrames, Cypress doesn't like iFrames. You can work with them with proper plugins, but they are always problematic. There were also some aspects that Cypress, for example, would need due to the size of our web page. We, for example, needed a whole page screen. When something failed, usually you wouldn't have it on the screen because it was checking elements that were under the screen.
And the way how scrolling works in Cypress, you won't always have the proper position of the screen on your error. Because, I don't want to go into technical details too much because probably some of your fans know more than me. But basically, Cypress is properly scrolling to that place, but it doesn't mean that on the, let's say, error view, you will have the scroll down. And that was a big problem.
And if you want to set up the proper way of scrolling the page, you can set up something like full screen error. It doesn't always work. There is, maybe now it's fixed, I haven't checked it in some time because we were doing the work around a year ago.
There was a known issue, which back then was kind of won't fix, that if you're using React with specific configuration, if you try to do full screen page, like, you know, the whole page that is very long, it will be repeating always the first view. So you will have, let's say, seven times the header and just it. And, well, for one, it's a problem for us. It was a problem for others. It's something that they can live with.
We had some workarounds around it, but, you know, we had the workaround on workaround, workaround. And the point that kind of was for us, that broke the camel back, was the situation with the currents. I don't know how much you know about how the Cypress parallelization works. Oh, tell me, please. So basically, in Selenium 10 times, you basically were completely on, you could do everything. You had the Selenium grid, which you could self-host and you could do your own parallelizations.
There were companies and they were still like BrowserStack, SauceLab, that they were providing your Selenium grid options. And Cypress kind of decided to make business on selling parallelization. So framework by its own, it's free to use. You can create your own plugin to do local parallelization, but kind of, and now it's much more popular. Back then, there wasn't many of such options. One of them was Cypress.
Don't buy the, I'm sorry, I don't remember his name and he's going to be mad at me because we're talking quite a lot. You know, this is typical blanking during the interview. So Andrew, sorry, Andrew, that I couldn't remember your name. So Andrew that would own Cypress also had the currents. His implementation that you can buy and you can use for, you know, with their hosting, that kind of stuff. Kind of similar to the Cypress cloud.
The difference is that in the amount of users that you would need for Cypress cloud for organization like us, it was in Cypress cloud, it was 10 to 20 times much more expensive than using the currents. We wanted to discuss it with Cypress because basically at that time they had packages that was like, you know, you have the number of users and number of runs. We wanted more users.
We didn't have problem with number of runs, but we wanted more users and they wanted from us to go to the much bigger packet, which had much more runs, which you couldn't use. But they weren't very flexible with negotiating that kind of situation. And, you know, that was more than basically my whole QA budget for that year. So we went with currents and at some point Cypress was very mad at currents, calling them bad, not enemies, bad competition. Because let's not go into the technical details.
I am on both sides. I understand the point of view of the Cypress because as much as it was funny for me, the fact that the library that was used by currents was called Cypress cloud was very on the nose and kind of could infringe on the naming of Cypress. But there was a lot of bad blood there to the point that Cypress blocked the ability to use currents. If you have the Cypress above, I think version 13, you cannot use it with currents.
And for us, going with Cypress wasn't the option with the Cypress cloud. It was too expensive and we had lots of other smaller issues and we were seeing lots of the problems. Plus, this kind of move showed us that we couldn't treat Cypress as, you know, a serious partner in any form of negotiation. So we bought, we're currently still with the currents on our Cypress that is not updated because we are, I think, on version 12 something, the last that is supported by currents.
But at that point we decided we will be leaving the Cypress ecosystem. That was just to show you how long it took. That was September 2023. So just to show you that the process wasn't fast, it was quite long. Because for first thing, most of my team is doing typical day to day business work, testing stuff that is regular, still maintaining our regular test. So even if that was one of our goals for 2024, it wasn't, you know, as easy.
It wasn't something that we knew that's going to happen from day to day. So, yeah. So you have no background. Do you have any question to the background? Yeah, no, that's a good introduction to the topic. And if you decide such a decision to change the test automation framework from one big player to another, you have to deal with a lot of things. You have to mention all the transition stuff and as you said, you have all your day to day business every day.
So what was your plan or how did you did the transition there to your new framework? So transition is actually happening because first we had to say one thing, selecting the framework. Because at some point we kind of all expected that Playwright may be the framework that will win. But actually we, I'm not sure if I'm here. Oh yeah, you're connected because I had something that like stopped recording. But I think I'm back. So again, so Playwright seemed as possible the best candidate.
But actually we didn't know that. We were suspecting it, but you know how it is with that thing. So first thing what happened, we had to look what are we using the Cypress? What are the things that we like about Cypress? What are the things that are not working for us? And then from that we went to kind of more generic. What do we want from our end-to-end framework? What is our approach to testing? To create kind of our list of maybe not goals, but the aspects that are important to us.
And next step was looking. There was around, I think at that point, because it may seem like a small part of time, but we had the first iteration of our goals done in the first quarter. Then there were quite a lot of other projects happening. So the second quarter of last year, we couldn't do much with that project. And around, I think it was July, we started to looking for the tools. We're looking at different tools. At that moment, it was mostly static.
So we had calls with some companies looking at the tools that are on the market. And I'm going to say I was kind of downhearted. I spent lots of my career doing research on tools. I can use company names on your show or I should avoid them? Oh, please avoid it. Okay. So when I was working at a company that is no longer on the name that I was working there, basically I was a Guild Leader of this automation, kind of in the Spotify model, where the Guild is more like community of practice.
So and one of my jobs was actually doing research, what tools are there and what are available. And back then I was spending a lot of time on this. And that was 2019. And when we started doing it in 2024, it was like nothing has changed. There is nothing new. The last big thing that came to the market was Playwright, which was at the beginning of pandemic. And lots of things that were at that time looking like promising thing, later died or changed into something else.
And that kind of broke my heart because I was expecting to see such much progress. If you asked me if I was doing such positions five years ago, I was expecting that, you know, lots of tools that are record and play right now, they will be the big part of our market. That most small companies or even big companies for the fast development, they won't be doing code development. They will be using record and play tools.
There are tools like that. Yes, there is Test Team, there is Map, there is Polish Backpack.io. But they all are very limited and you have to have very specific needs to be able to use that kind of tools in your process. So we kind of spent some time on the research. We had a big table with different aspects we were checking, which spent what. We were giving them points. Some criteria were much more point-worthy, others less.
And we have to select something like, I think, seven tools at the end that we decided to go into actual hackathon. Actually, not just looking statically what they have on the page, what are their demos, but actually doing proof of concept on our site. And I think that was, I don't have the timeline next to me, so I cannot just check.
But I think around September, we, as the quality engineers, with help of interested parties, few backend developers, two frontend developers, decided to finally go, to finally we had time to do the actual experimentation of the tool. And as you can expect, Playwright basically left everything else in the dust. I'm not going to use the names of the tools, but one of the coded tools that I was actually, because we had more people than tools and some tools have few people doing them.
One tool was done by me and some backend developer that he is, well, he is backend developer, but he knows frontend other stuff. And we both couldn't set up one test. We actually spent whole day and we couldn't make actually working test going with the documentation and quick setup guide.
It was later we found out that they had some big update of, you know, the framework, something like few weeks before, but the documentation was so out of date that even trying to, you know, we actually had to go to the issues on the GitHub repository
to, you know, to do that kind of stuff. And that was also already big no-no for us, because if you have to spend your time during your daily development of the tests on basically not finding information on, you know, documentation, but actually had to dig through the support tickets, it's something not right.
But I think the most interesting case for me, I kind of mentioned it to you, I think in November when we were on the retreat, one of developers got tool that now had some AI component for creating test. And he basically in the language, a la BDD, had to, you know, describe what the test was supposed to do. So something like go to the, you know, homepage, select this category, no, select the first product you see on the page,
add product to cart, buy it. Basically, because what I forgot to mention, we had a list of, I think, 10 tests prepared to see how many tests each of us would be able to write and to have, you know, baseline to compare, because we don't want everybody to write completely different tests. So we kind of had this one. And he's showing us, he's calling us in the middle of the day because he's saying either I'm too stupid or I don't understand what's happening.
He show us the, you know, the script he wrote. Basically, he did copy paste of what we had prepared in, you know, the words for that system to write. And that tool decided to, you know, scan our page. So we are seeing the recording how it's actually, you know, automating it. It went to the homepage. Then it went to not the product. You had thousands of our products on the homepage because, you know, we have the presentation of different plates
that you can buy. It went to some of our menus, went to some collection, then went to another collection, then to another collection, then to some brand page. One interesting thing what happened on the way, it managed to deal with the pop-up because some pop-up about newsletter or something else opened and it automatically closed it, going further into the site. And finally, after six minutes of clicking, it went on the product page. It added it and it finally added it to cart.
And, you know, looking at the scenario, you are like, there is nothing like this on this tool. I later talked to someone that actually is using this tool, not the AI part, but the base tool. And he told me that this is something connected how they're using something, some AI agent like, you know, chatGPT. And if probably you would try it to do after hour, it would generate that test completely differently because what it's doing is first using
apparently those AI to create the path. And if you accept that path, it writes it as, I don't remember, I think it was the puppeteer script under the hood, but don't quote me on that. And probably if until we approve that script, if after hour it would try, the large language model that is used for the generation of the test probably would generate a completely different path for that. Sounds very interesting hackathon you had on this stage.
Yeah, I'm actually one thing that I wish we had done, we would record that because I see it in my own eyes. I even tried to reproduce it later using that tool. I had different stories connected to that part, but I couldn't reproduce it in such a funny way as it was doing it back then.
So it's very interesting to hear that you did such a day with all your QA guys and development guys to choose from these tools for test automation framework, because choosing a tool is always a hard thing and to do it with real, your test cases with 10 scenarios and try it in different tools on one day is, I think, a very, very smart way to check if the tool fits your requirements.
But you know, what is the worst part about it? Then explain to your boss why after doing all this research you choose still the tool that is the most popular tool and was the obvious choice from the beginning. But then you have the proof, yeah? Yes, and of course, to make it even fun, we did lots of research. We actually checked lots of different things about Playwright and I'm going to tell also the story about how Playwright performed at that workshop
because it kind of went a very good pairing. Our developer that is currently our expert on using AI to write code, he's even doing internal trainings for us and he was playing with the cursor and using the also Playwright, just now I'm blanking on the name. I think it's called Inspector for doing the record and play, to record the test and it's generating your code. He actually managed to write all the 10 cases and stabilize them to be, you know, actually stable during the hackathon.
Okay, great, yeah. For comparison, the only person that was able to also do the 10 tests was someone that was doing strict record and play. So, actually it was far away the best tool and the best results and it was, I think in the hands of different person it wouldn't have went so well because, you know, this is the kind of problem with such a small sample size. But it also was showing the potential of the Playwright.
Of course, what is worth mentioning is that those tests that were generated back then wasn't what you would call might enable. So, if nothing changed, probably you would have to basically record the whole test again similarly how you do it with record and play because the code wasn't done as very might enable. It was basically done to show it's possible.
Sounds good. Did all the guys who did this challenge with you, this hackathon, follow this decision with Playwright or was there a hard discussion because someone wanted his tool? No, actually there were a few tools that were performing quite well. But at the end of the day, we were doing a demo, but basically everybody was showing what they were using,
what is their opinion. After seeing the Playwright in action, it was quite clear that there are a few good options, but nothing that will compete with Playwright. So, you decided to go to Playwright and now you have your big repository on old test cases in Cyprus. And now, what was your strategy to go to the new framework? The actual strategy is, I need to go later with our approach to end-to-end tests. We have something that is called, we have our full regression that is running not nightly.
It's running a few times a day, basically. It's running at the start of the day, at the middle of the day, and basically at the evening because we kind of want to get some good information what is happening during changes done by developers. Because full regression has a lot of tests that, if something will be broken for a few hours, in our case, those tests are not like something that will kill us. It's good for fixing them as soon as possible, but it's not critical.
We have the smoked suit that is running after every deployment. And this suit is something that actually, if it finds bugs, usually it's something that means either you are fixing it right now or it should be rolled back. So, actually the properly critical thing. And we have something that we call top 10. Those are the 10 most critical paths in our system. Those are the money makers. The things that something is broken there, basically drop everything and go fix it.
And those are running regularly on our production environment. Something like during the high season, like Black Friday, so when there is lots of money to be made, they're running every 10 minutes. Outside of high season, they are running every half hour. And we decided to start with those because those are the tests that need to be the fastest. Those are usually not the easiest tests to write, but they are also kind of vertical stylized of our system.
So, the first thing that is currently happening, because we haven't finished it, the goal was to finish it until the end of December, but due to a few problems, we didn't manage it, was actually the movement of those tests to the playwright. The plan was, it's kind of evolving, but move those tests, set up it as the main tests that are running every 10 minutes. And then as the new functionality will be coming, we'll be writing that test just in playwright.
And in the meantime, we'll be slowly moving all the tests. So, if something will be broken in Cypress, if it's a small thing like change of locator, you will be fixing it in Cypress. But if it will be change of logic, the proper way is to move the test to playwright. But we are still not at this stage because the top 10 was, I think, the last test we are moving finally this week. The week of the recording, not the week of the premiere of the episode. So, as you mentioned, we started moving it.
But of course, even after doing all of the research, there were coming a lot of surprises to us. First thing that actually caught us quite off guard was the way how the session is managed in Cypress versus the playwright. Because in our case, we have a lot of A/B tests on our site. And we have a lot of additional things that we are keeping in local storage, session storage, and cache, and the cookies. So, there is a lot of things that we need to set up for our test to work.
Because, again, we are doing lots of testing on the production, not only the test environments, but also on production. I am a big fan of the saying that I don't remember who told me that, but there is no second testing environment such close to production like production itself. Yeah. So, basically, if something can be tested on production and it's not a risk to break something for thousands of users, I'm usually a fan of testing it on production.
So, since that, we had to set up our test to actually first cut off the analytical cookies. Because if we don't want to do that, this means that our test will be actually collected in analytical data. It was even a funny story. I think that -- I don't have that plate behind me. There was one plate for semi-popular author on our site that one of our testers very liked. So, it was used in a lot of tests that we're doing.
So, you could imagine that daily our automation could go on that page, I think, maybe hundreds, thousands of times. And being in the kitchen all the time, I'm hearing that maybe they should go to that guy, talk about some promo, because a lot of people are visiting his pages and not buying it. Because, you know, in stats, we were showing that we were going to that plate. So, again, this is old story.
When we were still figuring out where we should be informing, what information should be in the test to not affect our sales data and that kind of stuff. But now we have this information. We have this all figured out in Cypress. And now it's time to figure it out in Playwright.
And it starts to be fun, because at some point, nobody realized that, wait, but Cypress has very good ways of doing the interception of the API request, because we are doing lots of things with the API request for setup and copy, cookie, and that kind of stuff. Cypress Playwright also has the options for that.
But actually, I think the way how it was done in Cypress was basically much better, because in Cypress, we had, I think, maybe two liners for, you know, intercepting the cookie, not the cookie, the request with the, let's say, authorization. It was an authorization, but let's say for sure that it was authorization.
And in case of the Playwright, actually, we had to spend some time looking at the different documentation, the different suggestions, how to properly set up the way how we're handling the setup of both the requests, how to properly set up the, oh, yeah, block hosts. Because in Cypress, it's easy. You have basically in the config option block host, and all those hosts are blocked. In Playwright, as far as we know, there is option that we basically missed something.
It doesn't have such functionality. We had to write it on our own, so we had to find how to properly intercept the request and how to do it in the way that it would be always working there. So we had some mechanics for that, but it was like, it wasn't even in our, you know, in our spreadsheet of things that the framework should have, because we assumed that if we have, it can handle requests, it will be able to handle that kind of thing. So it wasn't a separate position in this.
And that was kind of thing that we are still visiting, because you have one thing like that, then you come with another thing. And then something that actually was for me very funny, old tests. Because lots of tests were written a long ago. I think one of the tests that we're moving, it's older than I was in the company. And at start, kind of we had that decision made, but it was forgotten. We tried to, you know, rewrite the test. So we wasn't doing it one to one.
So we are looking at the logic, but we shouldn't be taking much of the code. But at some point it was lost. And people were trying to, you know, kind of rewrite Cypress code in Playwright. So it's like trying to put, you know, the square back into circles. So this is, I'm not going to say that it's kind of fault of anyone. This is, you know, just basically standard work in the pressure. The deadline is coming. You're starting to, you know, to think differently about stuff.
You still are not fully trained in the new tool, because we did some training, but you know how it is. You still are learning how to use. So a lot of that kind of stuff is coming. How it's called. The data management is done. A lot of that kind of stuff that you think all frameworks are doing it kind of the same. No, they are not doing the same. Yeah. So you're dealing with some surprises on your transition way. But what is your strategy?
So if you, I think before you started the transition, all your test automation engineers had a lot of work to do anywhere the whole day. So how do you deal with creating new test cases and make a transition of the old ones? How do you deal with that? Actually, we were kind of happy. The reason why we decided to use most of December for moving the tests, for doing the setup of the top 10 is that it's holiday season. For us, it means that there wasn't as much feature development in component.
There was a lot of cleaning, a lot of doing that kind of stuff. Because as with every e-commerce, the kind of moment from a few weeks before Black Friday to Christmas is the moment that the companies are doing most of money. So you don't want to put any big feature outside because you don't want to lose that time window of selling for possible issues. So that was one of the things. The other thing is basically prioritization. We are smart team, QA wise.
Because most of the developers are doing their own testing, we don't have a tester per team. We are kind of working in a similar way to DevOps team. We are a platform team that is giving support. We're doing research into, for example, how to test payments in different markets. Let's say we are going to start using some payment method in America and the team has no idea how to test that. So usually they come to us and figure out how we can test that.
Or they come to us, the situation is like that, we have no idea how to make sure this will be tested. And that's why we kind of have more time to do that kind of maintenance stuff. But it depends. I'm already seeing the roadmap for the next few months. I would expect that probably in February, March, there won't be much time on the automation. Right now, we're kind of doing rotation. The two people are supporting teams with their issue.
One is writing automation and we're doing rotation that everybody can do some automation tasks. But I don't think it will be possible for the February and March. Right now, I'm kind of trying to still work at the team capacity, have some for that task. But we will be seeing how it actually will be working when the time will come. So my strategy is kind of looking at the backlog and having some more capacity frozen for that. That's great.
And if you look at the upcoming year 2025, do you think you will manage the whole transition this year? Or is it not possible? Or is it just not relevant for you because it goes like it goes in the flow? Actually, that is a very hard question. Personally, I would hope we will manage to move more stairs in 2025 to play it out. Because what actually was supposed to happen last year, but didn't because we knew we would be leaving Cypress, is get more front-end developers into test automation.
It's actually happening with Playwright right now. Developers are participating in the pull requests. Until we finish, you know, stabilization, the top 10, we don't want them to write tests yet. But we will be planning to give them the ability to write, maintain their own tests. So that will be also fascinating for us. And this is giving me hope. The other thing that is giving me hope is an experiment done also by that developer I mentioned before.
He did an experiment that is still not working properly, but with some maybe a few more tries it will be. Using the cursor, which is, you know, the IDE we've built in AI for translating tests from Cypress to Playwright. And he did a few examples on his own. Those were great. But when he wanted to do the demo for us, it didn't work. It actually on the demo, as you know, didn't work. But I'm seeing the potential there. Maybe it's either we need to create better prompts or something else.
But basically what he did, he took our Cypress repo, the copy paste inside the Playwright repo, and then he discussed it with the AI agent. So this is the Cypress repo. This is the documentation. Here is the part that is Playwright. And now I want you to help me. The test is, for example, on the homepage, smoke test or something like that. And I want you to create for me the version of that test in Playwright. And after a few tries, it managed to do something that actually worked.
My problem with it is I'm still not convinced about the maintainability of the test. But I'm seeing it more as we learn probably more the ability of Cursor as the whole team. I'm expecting that the speed of writing those tests will go faster. Yeah, great. So maybe we can make a future episode about this, about the experience with your final migration and transition to Playwright with AI support.
If you want, I have one interesting tidbit, because I was playing with the cloud desktop, the cloud, you know, the AI agent, the desktop agent that you can control your desktop. And I was doing some experimentation how to use it to test automation. Okay. Okay. So maybe we have a future topic here. Yeah. Maciej, thank you very much for all these insights. I think it was a very, very practical hands-on episode.
I think for a lot of test automation engineers out there to get an idea of what to think about the hackathon and the transition from one tool to another, and your thoughts about why you're choosing which tool and why we're leaving. That was very interesting for me, too. And yes, I wish you the best for the rest of your transition and for this whole year. And we will see in person in October or November, I think again. October. I think it's October this year. Okay. So a little bit up there.
And yeah, I'm happy to see you then again. And thank you for joining the show. Thank you. Bye. Bye. Bye. Bye. Bye. Bye.