Crawl budget explained

Crawl budget is the number of URLs Googlebot will fetch on your site in a given period. Google itself frames it as two forces: the crawl rate limit, how hard it can hit your server without slowing it down, and crawl demand, how much it actually wants your pages based on their popularity and freshness. Multiply the two and you get a practical ceiling on how much of your site gets crawled.

Do you even need to worry about it?

For most small sites, no. If your store has a few hundred URLs, Google will happily crawl all of them and crawl budget is not your bottleneck; content and links are. The topic becomes real when a site is large or generates URLs faster than it publishes worthwhile pages. That usually means one of these:

• Catalogs with tens of thousands of products or more
• Stores where faceted navigation spawns endless filter and sort combinations
• Sites with session IDs, tracking parameters or infinite calendars in their URLs
• Frequent product turnover where new and updated pages need to be found quickly

Where the budget leaks

When Googlebot spends its allotment on the wrong URLs, the symptom is slow indexing of the pages you care about: a new collection sits undiscovered for weeks, or price and stock changes take too long to surface. The waste almost always comes from the same places, low-value URLs that Google keeps re-fetching:

• Parameter URLs from filters, sorting and pagination
• Faceted combinations that duplicate the parent category
• Soft 404s and thin pages that return 200 instead of a proper status
• Long redirect chains that burn a fetch for every hop
• Internal links pointing at redirecting or dead URLs

How to steer Googlebot deliberately

The goal is not to make Google crawl more; it is to make it crawl the right things. A handful of levers do most of the work.

Cut the low-value URLs first

Use robots.txt to stop crawling of parameter patterns that never need to be indexed, and return honest status codes, real 404s or 410s for gone products, so Google stops revisiting them. This overlaps heavily with your robots.txt and sitemap setup, which is the single highest-leverage place to shape what gets crawled.

Point the way with clean signals

• Keep an XML sitemap that lists only canonical, indexable, 200-status URLs, no redirects, no noindex, no parameters
• Flatten redirect chains so any old URL points directly to its final destination in one hop
• Fix internal links so they target the final URL, not a redirect
• Strengthen internal linking to priority pages, because crawl demand follows the links and authority you send inward

Measuring it honestly

You cannot manage crawl budget from guesswork. Two sources tell the truth. The Crawl Stats report in Search Console shows total requests over time, response codes and which file types Google fetches. Server log analysis goes further: it shows exactly which URLs Googlebot requested, how often, and how much of that effort landed on parameter junk versus real products. If a large share of Googlebot hits are on filtered URLs, you have found your leak. This kind of log work sits at the core of our technical SEO service.

Treat crawl budget as plumbing, not a growth tactic. Fixing it will not, on its own, lift a page that does not deserve to rank. What it does is remove the friction between publishing something good and Google actually seeing it, which on a large store can be the difference between ranking this month and ranking next quarter.