Crawl budget is the number of pages a search engine like Google is willing and able to crawl on a site within a given period of time, determined by the site’s size and speed, its popularity, and update frequency.
On large sites (eCommerce with thousands of products, publishers with extensive archives), efficiently managing crawl budget – by blocking low-value pages in robots.txt, eliminating redirect chains, and optimizing internal link structure – is essential to ensure important pages are discovered and reindexed quickly.
Why it matters
For small sites with a few hundred pages, crawl budget is rarely an issue; for large sites, poor management can lead to new or updated pages not being crawled and indexed for weeks.