SEO Atlas
Crawling IndexingSpidering

Crawling

The process by which a search engine discovers URLs and downloads their content using automated software, before any indexing or ranking happens.

In full

A crawler starts from known URLs (sitemaps, previously crawled pages, links) and follows references to discover more. Each fetch is subject to robots.txt rules, host load limits and the crawler's own scheduling priorities. Crawling only obtains bytes; whether those bytes are stored and made retrievable is a separate decision made during indexing. A page that is never crawled can still be indexed URL-only if it is linked to, but Google will have no content to describe it with.

Example

A news site publishes a new article, pings its sitemap, and Googlebot fetches the URL within minutes; the article is crawled but does not appear in search until it is also indexed.

Related terms

Indexing

Storing and organising a crawled page in a search engine's index so it becomes eligible to be retrieved and ranked for queries.

Googlebot

Google's main web crawler, which fetches pages for Google Search using two user agents: Googlebot Smartphone and Googlebot Desktop.

robots.txt

A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.

Crawl budget

The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.

XML sitemap

An XML file listing URLs a site wants crawled, optionally with lastmod dates, used as a discovery aid by Google, Bing and other engines.

Where to read more

Tools that touch this

advertoolsBotifyCrawleeJetOctopusKatanaLibreCrawl Technical SEO Audit MCP