Importance of growing crawl budget (James Dooley Interviews Andrew Halliday)

James Dooley: Hi, today I’m joined with Andrew Halliday, who is an absolute legend when it comes to technical SEO. Today’s topic is the importance of increasing and improving crawl budget. Andrew, explain why improving crawl budget matters for websites. Andrew Halliday: Every website starts with a limited crawl budget. There is no fixed number, but let’s say a site gets around 5,000 crawls per month. If that site has 10,000 pages, Google is not going to visit every page every month. That means changes you make are slower to appear in the search results. The more crawl budget you have, the faster Google re-crawls your pages, reflects updates in the SERPs, and supports ranking growth. James Dooley: If we assume that baseline of around 5,000 crawls, how do you actually increase crawl budget? Andrew Halliday: There are several ways. The first is stability. If your site goes down or throws 500 errors, Google will reduce crawl activity very quickly. That is one of the biggest red flags. Foundations come first. After that, it is about trust signals. High quality backlinks help. Schema helps Google understand your business. Legitimate traffic helps too, whether that is paid traffic or real social traffic. All of these signals encourage Google to crawl more often. James Dooley: Does content quality affect crawl budget as well? Andrew Halliday: Yes, absolutely. If you are publishing low quality content, especially unedited AI content, Google will see no value in revisiting your site. That reduces crawl frequency. AI can be used, but it needs editing and quality control. Poor quality content tells Google there is no reason to come back. James Dooley: How do you track whether crawl budget is increasing or decreasing? Andrew Halliday: You need server log files. That is the only reliable source. You should analyse how often Googlebot hits your site over time. Looking at a single day is pointless. You need weekly and monthly trends. You can download logs and load them into tools like Screaming Frog Log File Analyser. There are also other tools, but logs are the source of truth. James Dooley: If someone wanted to outsource server log analysis to a specialist like yourself, how does that work? Andrew Halliday: They can contact me through my main site or my technical SEO site. I normally ask for three months of server logs. Six months is even better. Anything less than a month limits what you can see, but you can still spot major issues like 500 errors. Logs also show wasted crawl budget, such as Google hitting 301 redirects repeatedly instead of clean URLs. James Dooley: If someone cannot afford professional log analysis and wants to use tools themselves, should they only focus on Googlebot activity? Andrew Halliday: Yes. Other bots matter, but if you want to understand SEO performance, Googlebot is the priority. That said, there are different Google bots. Newsbot, Jobsbot, and others carry higher trust signals. Triggering those bots through things like Google News or job listings can increase overall crawl activity. James Dooley: For people who are not tracking crawl budget at all, how important is it really? Andrew Halliday: It depends on site size. For a small local business with five pages, it barely matters. For medium to large sites, especially ecommerce sites with tens of thousands of pages, it is critical. Crawl budget directly affects indexation, updates, and rankings at scale. James Dooley: Final takeaway for anyone watching? Andrew Halliday: If you run a medium or large site, you must track crawl data. Fix 500 errors, remove unnecessary redirects, and monitor trends in Googlebot activity. Crawl budget is a foundational SEO metric, even if most people ignore it. James Dooley: Thanks for joining, Andrew Halliday. It’s been a pleasure. Andrew Halliday: Thank you.