Technical SEO
X-Robots-Tag
What is X-Robots-Tag?
The X-Robots-Tag is an HTTP response header that gives search engines indexing instructions, such as noindex, nofollow, noarchive or nosnippet, in the server's response rather than in the page's HTML. It supports the same directives as the robots meta tag.
Because it lives in the header, it works for files that cannot contain a meta tag, such as PDFs, images and other downloads, and it can be applied in bulk through server or CDN configuration, for example to every URL in a directory or every file of a certain type.
Why it matters for SEO
The X-Robots-Tag is the only way to keep non-HTML files out of the index, and a convenient way to apply rules site-wide. It is also a common hidden cause of indexing problems: a stray noindex header left over from a staging environment can remove pages from Google while the HTML looks perfectly fine.
Example
A software company wants its downloadable PDF price sheets kept out of search results so visitors land on the pricing page instead. Its server adds the header "X-Robots-Tag: noindex" to every response for files ending in .pdf. An HTTP header checker confirms the header is present on the PDFs and absent on HTML pages.
Free tools for X-Robots-Tag
More Technical SEO terms
All Technical SEO termsPut the theory to work
LazySEO researches keywords, writes SEO articles and publishes them to your site on a schedule.