When is A/B Testing NOT A/B Testing?

I've been in SEO and online marketing for over 20 years and I'm here to share the wealth of knowledge , hints and tips I've amassed over that time . Hi , welcome back to SEO is not that hard . It's me here at Dawson hosting , as usual , and today I just wanted to talk , well , pose a question when is AB testing not AB testing ?

Now , this has come up to me because there's a tool out there which has started to claim that they can do AB testing . So if you were using using this tool , you can go to it and say I've started a test on this day , I changed some content on this date , and then they will monitor your google search console and traffic .

I think they do it for google search console anyway , um , for before and after that , you know um change , so how your content was before you did that change and then how your content performed after that change , and they're calling this an A-B test . Now I just want to cover why it's not an A-B test , it's actually a before and after testing , okay .

So obviously some people out there don't appreciate the difference between the two . I think it's important to know the difference between the two . Now , I'm not saying that what that tool is doing is bad . We kind of do the same thing on keywords people use , but we don't call it A-B testing because it's not A-B testing . Okay , it is before and after testing .

So let's start with before and after testing . So this is probably the easier of the two methods to understand and certainly to implement .

So essentially , what you do is you make a change to your web page , you update the headline or you redesign a button or you do any kind of change on that page , and then you compare performance metrics like clicks through rate or conversions or the number of clicks from before the change and then after the change .

So , for example , let's say you change a page , you add some text to it and add an image , and then you notice that next week your conversions increased by 20% the following week , which is great , right ? Well , possibly . But the problem with before and after testing is it doesn't account for external factors that could could influence your results .

So here are a few those factors that can change . So seasonality , so your after data might . You might be collecting that during a holiday period or a major event in your industry could be traffic changes . You know you could have run an ad campaign or you could have expired .

Experience of spike in organic traffic for a completely unrelated reason , like someone linked to you . It was a newsworthy item , something like that . Or it could be an algorithm update . Google's done a rolled out core update or one of its many other algorithm updates . That's affected how that piece of content ranks .

That is completely unrelated to any change you made on that site . So before and after testing assumes that the only variable that impacts your results is the changes you made .

In reality , the digital landscape is so full of noise , it's so full of other variables that aren't just the content or the design of the page that it means that it's only a very , very basic indicator of whether you've done well or not . You can't fully give all the weight and the glory essentially to the changes you made . So let's look at A-B testing .

Okay , so this is very contrasting . It's sometimes called split testing . So in an ab test you create two versions of the web page . I've covered . I think I've done a podcast all about ab testing before , so if you want to learn more , just have a search the back catalog for them .

But essentially you create two versions of a page a version a , which is the original that's what the page already had Then a version B the variation with the new changes in and you then show these versions to different segments of your audience .

So with A-B testing , if you've got two versions written at the same time , you can essentially work out if the change that you made has a positive effect on how people interact with the page . Now the trouble when it comes to saying has the change given you a positive ranking boost ?

It's impossible to do at the page level , on an individual page , because Google will only rank one version of the page at a time . It won't have two different versions ranking and show two different versions in the search engine results for you . Now there is a way around this .

If you've got a site with , you've got templates with lots of common pages For example an e site , we've got lots of pages that are all product landing pages you can say you can say leave half of your pages with a current template and add half of your pages with a new template and then wait for google to index the ones with a new template and then see how

they perform as a group against the um , the core group that you didn't change . And you can do that , and there are some tools that help you do that , one being search pilot but it's hugely expensive . You're talking , I think , tens of thousands a month minimum spend for them and you know quite complex to do , not a simple thing to do .

So you can do that . But again , there are still variables in there that you can't completely qualify for , because you know you might have some pages in one set that have got some better internal back sort of better external backlinks than ones in the other set , and that kind of thing can skew it .

Um , but the essential thing of a b testing is that you know the test happens at the same time period against two different versions , so you can remove all the other variables like seasonality and things like that . So you know , in terms of what are the key differences between before and after testing and A-B testing is timing .

Before and after testing happens sequentially , so one after another . So you make a change , then compare the before and after data , whereas A-B testing happens simultaneously , which eliminates a whole group of external variables . Control groups Before and after testing has no control group . You compare it apples to oranges .

You know it's different time periods , different conditions . Whereas A-B testing includes a control group , version A , which allows for a direct comparison under the same conditions .

The reliability of results so before and after testing is prone to false positives or negatives , because it doesn't account for the factors influencing your metrics , whereas A-B testing provides statistically significant results , assuming you've got a large enough sample size and that you run the test for long enough .

Complexity Before and after testing it's very easy to set up . You don't need any special tools required , just make the change , publish it and you just look at your analytics from before and after and see how they perform over a time period . Whereas A-B testing requires special tools . You used to have Google Optimize . That's been sunset , you can't use it anymore .

But there's things like Optimizely or VWO to manage those split tests and split the traffic and analyse results . But there's a cost , both in terms of setup and in terms of running and operating those tools . So when should you use ? You know , before and after testing is actually quite useful .

You might not be able to split traffic for technical or logistical reasons . You're testing something really obvious like a big , dramatic change , like launching an Italian website design or changing a pricing model , those kind of things . Or you just need a rough idea of performance and you don't require statistical rigor .

So , for example , if your website isn't getting much traffic , then setting up an AV test probably isn't going to be worth the effort because the sample size is going to be too small to produce reliable results . Av testing is really good when you need precise , reliable insights about how specific impacts you know changes impact user behavior .

If you're testing incremental changes , like tweaking buttons and headlines , especially when it comes to conversion or optimization , you're tweaking to try and improve conversion , works really well and you have sufficient traffic to split between the two versions and get meaningful statistical data . So before and after is great . I don't want to knock it .

Let's say we use it when we're using keywords people use on their Google Search Console integration . We're making changes to content .

Content we use basically before and after , because there are a lot of times you change in a piece of content where it's not templatized , it's not hundreds of pages with that piece of content on it and all you can really do is a before and after test to see if the graph goes up or the graph goes down after making your changes .

And you have to kind of apply your own knowledge in terms of when google corruptors have happened , what your seasonality is , things like that . You've got to sort of overlay that yourself in your mental model and look at it . It's a completely fine thing to do , but I wouldn't call it ab testing because it's not ab testing . It's something very different .

Now , with both methods there's certain pitfalls you want to try and avoid , okay , so first all , don't rush to conclusions In both before and after testing . You know it's tempting to credit any performance improvement to a change but , like I say , correlation doesn't always mean causation .

So , you know , make sure you think about what other things are occurring at the time , like we just mentioned before , in A-B testing , you know , don't stop your test too soon . You need to let tests run long enough to reach a statistical significance , otherwise you're still just guessing , testing too many changes at once .

Whether you're doing before , after or A-B testing , keep it simple . If you test too many changes simultaneously , you'll never know which one drove results , ignoring audience segmentation . So in A-B testing , you've got to really make sure that your split is random and representative of your audience .

Otherwise , if you're sending people coming from one platform , say from social media , down to one version of people , coming from organic to another one , then yeah , these people are coming with very different intents and from very different starting points . If you don't randomize them and mix them up , then you could skew the results .

So if you've got questions about testing and how to test and how to analyze things , and do just get in touch , ask any questions always here to help and you know , if you found this show useful , if you found value , then please do consider giving us a review it really helps and subscribing again , that really helps so we can show up in your feed every week .

Um , and yeah , that's it for for today . So until next time , just keep optimizing , stay curious and remember SEO is not that hard when you understand the basics . Thanks for listening . It means a lot to me . This is where I get to remind you , where you can connect with me and my SEO tools and services .

You can find links to all the links I mentioned here in the show notes . Just remember , with all these places where I use my name , the Ed spelled with two d's . You can find me on linkedin and blue sky just search for ed dawson . On both . You can record a voice question to get answered on the podcast . The link is in the show notes .

You can try our seo intelligence platform keywords people use at keywords people usecom , where we can help you discover the questions and keywords people are asking online .

Post those questions and keywords into related groups so you know what content you need to build topical authority and finally , connect your Google Search Console account for your sites so we can crawl and understand your actual content , find what keywords you rank for and then help you optimize and continually refine your content and targeted , personalized advice to keep

your traffic growing . If you're interested in learning more about me personally or looking for dedicated consulting advice , then visit wwweddawsoncom . Bye for now and see you in the next episode of SEO is Not that Hard .