Meta Robots Directives

SEO

Meta robots directives are instructions placed in an individual page’s HTML <meta name=”robots”> tag (or in the HTTP X-Robots-Tag header), controlling indexing and link-following behavior for that specific page, using values such as noindex, nofollow, noarchive, or nosnippet.

Unlike robots.txt, which controls crawler access at the directory or page level before access, meta robots directives are read by the crawler after the page has been accessed, offering more granular control over post-crawling behavior.

Why it matters

A noindex directive accidentally applied to important pages can completely remove those pages from the search index, a frequent and serious error requiring regular verification for critical business pages.

← All terms in SEO Full glossary