Best of : Cloaking - a Blackhat story

Hello and welcome to . Seo is not that hard . I'm your host , ed Dawson , the founder of KeywordsPeopleUsecom , the place to find and organise the questions people ask online . I'm an SEO developer , affiliate marketer and entrepreneur . I've been building and monetising websites for over 20 years and I've bought and sold a few along the way .

I'm here to share with you the SEO knowledge , hints and tips I've built up over the years . Hello , welcome to episode 96 of SEO is not that hard , and today I'm going to be talking about cloaking . This is not something you might hear talked about by a lot of mainstream SEOs , because essentially , it's a black cat technique .

Now , I am not telling you this in any way suggesting that you do it this , um in any way suggesting that you do it . Um . It's interesting in terms of learning how certain black cat techniques work , because I think any rounded seo needs to know what to look out for if they come across this kind of behavior .

Or if you are dealing with someone who is suggesting that you do something and you need to know whether it's , whether it's black cat or not . And anyone who wants to do black cat stuff , knowing full well what the risks and the rewards are , then that's fine with me . I don't judge . It's not something I do anymore .

Um , I just talked about plenty of times because for me , I still believe that now I want to build for the long term . I'm building for the long term means not doing anything that's going to make overtly risky in terms of getting all my hard work destroyed by a google update or manual penalty or something like that .

So , with that disclaimer given , let's talk about cloaking . So what is cloaking in its simplest forms ? Well , that is that you essentially you show google one version of your website or web page and you show other people a different version . Why would you want to do this ?

For example , say , you want to rank something and you want to Google to think it's really nice and clean and nice , clean content . But it might not necessarily be content that's going to monetize very well or get people

to take an action that you want them to , that you might think Google might not like what the content of the actual page you want to show people is . So I'll show them some really clean , really clean page with really clean content and , hopefully , rank based on that content , because that's what Google and Googlebot will see .

Then when people click to me from the search engine result pages , they'll get a different version which gets them to do something completely different . Say , google obviously doesn't like this because it wants to show people in the search engine results based on what they're actually going to see when they land there .

That's essentially what cloaking is , so it's showing a different version of the site to Google than you show to everybody else . Now the story a black hat story if we go back to the years of 2008 , 2009 , this was before .

I was very clean in what I do seo and one of the things I was doing a lot of the time was buying expired domains to put content on to then link to mine and other people's websites . And you know it was not as black as it is now say it was more grey , but anyway , we knew that it was against terms of service .

But at the time Google just couldn't detect these kind of links . It wasn't sophisticated enough to detect that kind of link building . But it was one of those ones where if your competitors reported you and reported those links , google would hand out manual penalties to people .

So the key to doing that kind of work was you wanted to avoid a manual penalty so that you could just get on for the rest of your life and build the links like that . So we wanted to avoid being detected . How were people detecting links and reporting them back then ?

Well , they were using Clink reporting tools such as I mean , the main ones were at the time Yahoo . You could get quite good backlink information out of Yahoo and Moz . They had a backlink explorer and I think it was Majestic around back then . There was a few , but Yahoo and Moz were really the big ones .

So what we wanted to do was we wanted to keep our links and links we were building from this method out of those tools , but available to Google . So what we did was we cloaked our entire website . But rather than cloaking them against Google , we cloaked them against Yahoo and against Mars . I think it might have been SCMrush , the SEMrush bot .

There was quite a few ,

basically every bot other than Googlebot . We cloaked the site so we did it in a few ways . We gave different versions of the robotstxt . So Googlebot saw very clean robotstxt , that sort of allowed to go wherever it wanted , whereas we'd show everyone else a much more restrictive robotstxt .

But we also didn't trust that some crawlers and bots would respect our robotstxt . So we then would also cloak the entire site against any kind of bot . We also cloaked from the search engine results pages because obviously you could at the time like Google will give you some links .

If you type link in a page in the search engine results , it would report some of the links you knew about . So we also cloaked anyone that came from a Google search results page to look at one of our pages . We would remove the links . We'd keep the content there but we would remove the links .

So we essentially used cloaking Not to show Google something different to what we were showing most people , but it was just we're trying to show it , cloak it differently for those sites that could have found those links that would enable someone to have reported these networks or sites .

This was basically PBNs at the time , public blog networks we were creating and we used Cloaky in that way . On a technical level , the way Cloaking works is it relies on interrogating the user agent for any request that's made to your web server .

So you would use some software that would intercept every request that came to the web server and it would look at the user agent . And if that user agent was Googlebot , then we'd do one thing . If it was different kinds of other user user agents , we might make other decisions and again we'd also look at the referring url .

So if it was coming from google , then we knew that someone had clicked in google to find that , to come to this page , and we would then take actions depending on that . So you can , you can cloak , say , on any information that really comes in that in that header .

So that could be a referrer and it could be the user agent and would also look at the ip address , because obviously sometimes google will check a page , check a site , without using google as the header . Um , at least that's what we believed . You know I would have to have not .

It's been so long since I've done any of this stuff that I don't know whether google still do that or whether they did do that , whether it's just our fear that they did that , but we actually also built a database of genuine google ips .

So these are ips address addresses that we knew googlebot used , and we also shared that with other people who were doing similar things . So we we built up a database of IPs .

So if we ever saw Googlebot come from an IP and we used to do reverse proxy sorry , reverse IP lookups to see where they came from , because Google at the time , or probably still is was only coming from IP addresses that were registered in Mountain View in California .

So you could then go back and say , right , if any IP address that's come from Mountain View , we would treat it as if it was Googlebot or Google and would serve it that way . And this was our

aim was also , if we've got any kind of manual review , that we would show a manual user from Google . This is what they were getting . So that's how you actually do it . So how do you detect if you're suspicious that a site might be cloaking ? How can you as a genuine , normal person , check this ?

Well , again , it's going to depend on how sophisticated they've done it . I've just given you a whole details of how we did it , and we went to quite a sophisticated level . A lot of places go slightly less sophisticated . They might just look for Googlebot being the user agent and not do any of the IP checks .

And you can detect those by if you can get a plugin for your browser which allows you to change the user agent that Chrome sends when it makes requests to the web server , and they've got all sorts of choices like Bingbot , googlebot you can just do some manual ones , or you can enter your own user string and change the user agent .

So a lot of the time , if you find one of these , if you just change your user agent in Chrome to be Googlebot , you might find that a site would show you a different version and that's how you can spot how they do it .

But again , it all just depends on how far down the line of how sophisticated they've made their cloaking , how far they're going to try to detect whether it's true Googlebot or someone faking Googlebot before they'll actually show you the cloaked version .

So yeah , so I hope that's been a sort of useful explanation and , as I say , all this stuff is completely against Google's terms of service . It's against all the guidelines . If you do this , this and get caught , you will get banned to high heaven , um .

So I wouldn't suggest anyone does it , but I think it's important for people to understand how it does work , so they can , so that you can guard against it , so that you can see if anyone ever comes to you and says we've got this fantastic technique and this is what we'll do , you can just say , no , I don't want to do that at all and run for the high

offense . Or

if your risk reward says , hey , I want to try all the things , then you know at least you will know the risks around doing it . And obviously , just to reiterate , from my side , this is nothing that I've done since , uh , penguin , basically since penguin killed all this for me , and it's not anything I've done for for many , many years .

So you know , it's not a technique I'm using now and it's definitely not a technique that I recommend . So , yeah , I find it useful .

Before I go , I just want to let you know that if you'd like a personal demo of our tools at keywords people use that you can book a free , no obligation , one-on-one video call with me or I'll show you how we can help you level up your content by finding and answering the questions your audience actually have .

You can also ask me any SEO questions you have . You just need to go to keywordspeopleusecom slash demo where you can pick a time and date that suits you for us to catch up Once again . That's keywordspeopleusecom slash demo and you can also find that link in the show notes of today's episode . Hope to chat with you soon . Thanks for being a listener .

I really appreciate it . Please subscribe and share . It really helps . Seo is not that hard . It's brought to you by KeywordsPeopleUsecom , the place to find and organize the questions people ask online . See why thousands of people use us every day . Try it today for free at KeywordsPeopleUsecomcom to get an instant hit of more seo tips .

Then find the link to download a free copy of my 101 quick seo tips in the show notes of today's episode . If you want to get in touch , have any questions , I'd love

to hear from you . I'm at channel 5 on twitter . You can email me at podcast at keywords people usecom . Bye for now and see you in the next episode of seo is not that hard .