Google: No Need to Worry About Crawl Budget for some Site - Inanstech


Google’s Search Relations team says most sites don’t need to worry about crawl budget in the latest episode of the Search Off the Record podcast.

Gary Illyes of Google discussed the topic in length, saying the team has been pushing back on their previous messaging, and adding that a ‘substantial segment’ does have to care about it.

However, crawl budget shouldn’t be a concern for a majority of sites, Illyes explains:

“We’ve been pushing back on the crawl budget, historically, typically telling people that you don’t have to care about it.

And I stand my ground and I still say that most people don’t have to care about it. We do think that there is a substantial segment of the ecosystem that has to care about it.

…but I still believe that – I’m trying to reinforce this here – that the vast majority of the people don’t have to care about it.”


In an effort to clarify previous messaging, Google has recently been publishing more information about crawl budget.

For example, just last month Google dedicated a whole episode of its SEO Mythbusting YouTube series to the topic of crawl budget.

So who should care about crawl budget and who should not?

When to Care About Crawl Budget / When Not To Care

SEOs typically want to hear a hard number when it comes to crawl budget – such as your site has to have X-number of pages before crawl budget is a concern.

Continue Reading Below

But it doesn’t work like that, Illyes says:

“… well, it’s not quite like that. It’s like you can do stupid stuff on your site, and then Googlebot with start crawling like crazy.

Or you can do other kinds of stupid stuff, and then Googlebot will just stop crawling altogether.“

If forced to give a number, Illyes says roughly a million URLs is the baseline before a site owner really needs to care about crawl budget.

Sites with fewer than a million URLs do not have to care about crawl budget.

Read Also: website crawl budget explained.

Factors Affecting Crawl Budget

For sites with over a million URLs, these are some of the factors that could lead to, or indicate, crawl budget issues.

Factor 1: Pages not crawled in a long time

Get Your Personalized Recommendations In 3 Easy Steps.
Quickly Identify Areas To Optimize Your Microsoft Ads Campaigns With Our Free Report.


“What would I look at? Probably URLs that were never crawled. That’s a good indicator for how well discovered, how well crawled a site is…

So I would look at pages that were never crawled. For this you probably want to look at your server logs because that can give you the absolute truth.”

Factor 2: Widespread changes after long periods of time

“Then I would also look at the refresh rates. Like if you see that certain parts of the site were not refreshed for a long period of time, say months, and you did make changes to pages in that section, then you probably want to start thinking about crawl budget.”

Fixing Crawl Budget Issues

Illyes offers two suggesting for fixing crawl budget issues.

Continue Reading Below

First, try removing non-essential pages. Every page Googlebot has to crawl reduces the crawl budget for other pages.

So an excessive amount of “gibberish” content could lead to important content not getting crawled.

“Like if you remove, if you chop, if you prune from your site stuff that is perhaps less useful for users in general, then Googlebot will have time to focus on higher quality pages that are actually good for users.”

Illyes’ second suggestion is to avoid sending “back off” signals to Googlebot.

Back off signals are certain sever codes that will tell Googlebot to immediately stop crawling a site.

“If you send us back off signals, then that will influence Googlebot crawl. So if your servers can handle it, then you want to make sure that you don’t send us like 429, 50X status codes and that your server responds snappy, fast.”

Continue Reading Below

To hear more about the intricacies of crawl budget, listen to the podcast episode down below.





Blogging,10,SEO,6,Technology,1,Website Monetization,1,
Inans Tech: Google: No Need to Worry About Crawl Budget for some Site - Inanstech
Google: No Need to Worry About Crawl Budget for some Site - Inanstech
No Need to Worry About Crawl Budget for some Site - Inanstech
Inans Tech
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS CONTENT IS PREMIUM Please share to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy