Technical SEO
Crawl Budget
Last updated
What is Crawl Budget?
Crawl budget is the number of pages Googlebot (or any search engine crawler) will crawl on a website within a given timeframe. It is determined by two factors: crawl rate limit (how fast the crawler can crawl without overloading the server) and crawl demand (how frequently Google wants to recrawl pages based on their perceived value and change frequency). Sites with millions of pages need to actively manage their crawl budget.
For smaller sites with fewer than a few thousand pages, crawl budget is rarely a concern — Googlebot will typically crawl all important pages. For large e-commerce sites, news sites, or sites with extensive parameterized URLs, however, wasted crawl budget on low-value pages can mean important pages are crawled infrequently or not at all, delaying their indexing and ranking.
Why it matters for SEO
Optimizing crawl budget ensures that search engine crawlers spend their limited time on the pages that matter most for rankings. Poor crawl budget management — caused by thin content, redirect chains, or bloated URL spaces — can leave important pages under-crawled and underperforming in search.
Example
A marketplace with a few thousand products generates millions of URLs through filter combinations like ?colour=blue&size=9&sort=price. Googlebot spends much of its time on these near-duplicate pages, so new product pages take weeks to be crawled. Blocking the sort and filter parameters in robots.txt and linking only to clean category URLs lets crawlers focus on pages that matter.
Free tools for Crawl Budget
Go further with Crawl Budget
Put the theory to work
LazySEO researches keywords, writes SEO articles and publishes them to your site on a schedule.