On-page SEO
Robots Meta Tag
Last updated
What is Robots Meta Tag?
The robots meta tag is an HTML element placed in a page's `<head>` that provides page-level crawling and indexing instructions to search engine bots. Common directives include `index/noindex` (whether to add the page to the index) and `follow/nofollow` (whether to follow links on the page). Multiple directives can be combined, e.g., `<meta name="robots" content="noindex, nofollow">`.
Unlike the robots.txt file — which operates at the server level and controls crawl access — the robots meta tag operates at the page level and is only readable once a bot has already fetched the page. This distinction is important: if robots.txt blocks a URL, the bot never reads the meta tag. Specific bots like Googlebot can be targeted with bot-specific meta tags (e.g., `name="googlebot"`).
Why it matters for SEO
The robots meta tag gives granular, page-level control over how search engines index and link-follow behavior. Incorrect configurations — such as accidentally noindexing key pages — can cause significant ranking losses that are often difficult to diagnose without a technical audit.
Example
A news site wants article pages indexed but doesn't want Google to show long text snippets from paywalled stories. It uses <meta name="robots" content="index, follow, max-snippet:50">. On its internal search results pages it uses <meta name="robots" content="noindex, follow">, keeping those pages out of the index while still letting crawlers follow links to articles.
Free tools for Robots Meta Tag
Go further with Robots Meta Tag
Put the theory to work
LazySEO researches keywords, writes SEO articles and publishes them to your site on a schedule.