Technical SEO
Robots.txt
Last updated
What is Robots.txt?
The robots.txt file is a plain-text file placed at the root of a domain (e.g., `example.com/robots.txt`) that provides directives to search engine crawlers about which parts of the site they are allowed or disallowed from crawling. It uses the Robots Exclusion Protocol, with `User-agent` to target specific bots and `Disallow` or `Allow` directives to control access to paths.
Robots.txt controls crawl access, not indexing — a URL blocked by robots.txt can still appear in search results if other sites link to it, though its content cannot be crawled. For preventing indexing, the noindex meta tag is the correct tool. Common uses for robots.txt include blocking crawlers from staging environments, admin sections, duplicate parameter-based URLs, and internal search results.
Why it matters for SEO
Robots.txt is the first line of defense for controlling how search engines interact with a site. Misconfigured robots.txt files are among the most common causes of catastrophic SEO issues — a single mistaken Disallow directive can block an entire site from being crawled and indexed.
Example
A site's robots.txt contains "User-agent: *" followed by "Disallow: /cart/" and "Disallow: /search", plus a line pointing to the sitemap. Crawlers skip the shopping cart and internal search results but can reach everything else. A common mistake is a leftover "Disallow: /" copied from a staging site, which blocks the entire site from being crawled.
Free tools for Robots.txt
Go further with Robots.txt
Put the theory to work
LazySEO researches keywords, writes SEO articles and publishes them to your site on a schedule.