Crawl Budget
Crawl budget is the set of web addresses (URLs) on a site that Google can and wants to crawl, meaning visit and download. Google sets it from two things: how much crawling your server can take without slowing down, and how much Google wants your pages. Google says sites without lots of fast-changing pages need only a current sitemap.
Source: Google Crawling Infrastructure, Optimize your crawl budget (updated July 22, 2026), checked October 29, 2026
What Crawl Budget means for your business
Google's guide is written for big sites: over a million pages that change about weekly, over 10,000 that change daily, or sites where many pages sit in Search Console as "Discovered, currently not indexed." Google calls those rough numbers, not exact cutoffs. For everyone else, it says a current sitemap and a regular look at the Page Indexing report are enough.
Smaller sites can still waste crawling on addresses that shouldn't be there. Google's guide names endless scrolling pages, removed pages that still load instead of returning a 404 or 410 error, and long chains of redirects. It calls the list of URLs it knows about the part you control most.
Google counts each hostname as its own site, so www.example.com and shop.example.com get separate budgets, and all of Google's crawlers share one capacity limit. A slow server or a run of server errors makes Google crawl less. So if your shop runs on its own subdomain, we'd check its crawl stats in Search Console separately.
If new pages take weeks to show up in Google and you can't tell why, looking at how Google moves through your site is where we'd begin.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google's AI Overview cited Google's crawl budget guide and a 2017 Google blog post, plus SEO tool blogs, Moz and YouTube, and set a 1,000 to 10,000 page cutoff that Google's guide doesn't use. ChatGPT matched Google's two parts and cited nothing.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.