rate

Upcoming deprecation of Crawl Rate Limiter Tool in Search Console  |  Google Search Central Blog  |  Google for Developers

Friday, November 24, 2023 The crawl rate limiter tool in Search Console is being deprecated on Jan 8th, 2024. This tool has been available for over a decade, but with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated. Googlebot reacts to how the site–or more …

Upcoming deprecation of Crawl Rate Limiter Tool in Search Console  |  Google Search Central Blog  |  Google for Developers Read More »

Don’t use 403s or 404s for rate limiting  |  Google Search Central Blog  |  Google for Developers

Friday, February 17, 2023 Over the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate. The short version of this blog post is: please don’t do that; we have …

Don’t use 403s or 404s for rate limiting  |  Google Search Central Blog  |  Google for Developers Read More »