Robots.txt

SEO

Robots.txt is a text file placed at the root of a domain (e.g. example.com/robots.txt) that instructs search engine crawlers which sections of the site they are or aren’t allowed to access. The file uses simple directives like User-agent, Disallow, and Allow to control crawling behavior.

Important: robots.txt only controls crawl access, not indexing. A page blocked via robots.txt can still appear in search results (without descriptive content) if it’s linked from other sources, since Google knows it exists even if it can’t access its content.

Why it matters

A misconfigured robots.txt file can accidentally block entire sections of a site from indexing, directly impacting organic visibility. Periodically checking robots.txt is a standard technical SEO audit step.

← All terms in SEO Full glossary