The robots meta tag is an HTML element placed in a page’s <head> section, instructing search engines how to treat that page in terms of indexing and link following. Common values include “index, follow” (default), “noindex” (don’t index the page), “nofollow” (don’t follow the links on the page), and combinations thereof.
The difference from the robots.txt file is essential: robots.txt blocks crawler access to a page, while the robots meta tag allows the crawler to access the page but explicitly instructs it not to index it – which is why “noindex” is the correct method for permanently removing an already-indexed page from search results.
Why it matters
Incorrectly using robots.txt to block an already-indexed page won’t remove it from the index; the noindex meta tag is required for that purpose.