Crawl budget
Crawl budget is the number of URLs a search engine is willing and able to crawl on a site in a given period. It is set by the server's crawl capacity and by how much demand Google has for the site's pages.
Crawl budget describes how much attention Googlebot gives your site. Google explains it as the combination of two things:
- Crawl capacity limit: how many requests Google can make without overloading your server. Fast, error-free responses raise this limit; slow responses and 5xx errors lower it.
- Crawl demand: how much Google wants to crawl. Popular pages, pages that change often and URLs it has not seen before create more demand.
Together these determine how many URLs Googlebot fetches per day and which ones it prioritises.
Who should care. For most sites with a few thousand pages, crawl budget is not a real issue. Google itself says it mainly matters for very large sites (think hundreds of thousands to millions of URLs) or sites whose content changes very quickly, such as large webshops, marketplaces, classifieds and news publishers. If new pages on a small site are not indexed, the cause is usually quality or internal linking, not crawl budget.
What wastes crawl budget:
- Faceted navigation that creates millions of filter and sort combinations.
- Session IDs, tracking parameters and calendar pages that go on forever.
- Long redirect chains and lots of broken internal links.
- Duplicate versions of the same page (http and https, with and without trailing slash, uppercase variations).
- Soft 404s and low-value pages that are crawled repeatedly.
How to improve it:
- Block truly useless URL patterns in robots.txt, such as internal search results or endless filter combinations.
- Link internally to the canonical URL, never to parameter versions.
- Fix redirect chains and update internal links to point straight to the final URL.
- Keep XML sitemaps limited to indexable pages and use accurate lastmod dates.
- Improve server response times and keep error rates low.
- Return a 404 or 410 for pages that are truly gone so Google stops revisiting them.
Note that noindex does not save crawl budget: Google has to crawl a page to see the noindex tag. Canonical tags also do not stop crawling. Only robots.txt prevents the fetch.
Monitoring. The Crawl stats report in Google Search Console shows total requests, response codes, file types and the purpose of each crawl (discovery or refresh). Server logs give the complete picture and also show how much AI crawlers request from your server, which on big sites can be considerable.
Questions about Crawl budget
Does crawl budget matter for small websites?
Rarely. For sites with a few thousand URLs Google can usually crawl everything; indexing problems there are almost always about content quality or internal links.
Does noindex save crawl budget?
No. Google must crawl the page to read the noindex tag. Only blocking the URL in robots.txt stops it from being fetched.
How can I see my crawl budget?
Open Settings and then Crawl stats in Google Search Console to see daily crawl requests, response codes and response times. Server log files give a more complete view.