Robots.txt Directives

SEO

Robots.txt directives are rules written in the robots.txt text file, placed at a domain’s root, that instruct search engine crawlers which sections of the site may or may not be accessed for crawling, using directives such as Disallow, Allow, or Sitemap.

Robots.txt only controls crawling access, not indexing – a page blocked via Disallow can still appear in search results without a description if externally linked; completely excluding it requires a noindex tag the crawler can access.

Why it matters

An incorrect Disallow directive, accidentally applied to the entire site or critical sections like /blog/ or /products/, can block search engines from accessing valuable content, with severe impact on organic traffic.

← All terms in SEO Full glossary