Robots.txt
The robots.txt file is a plain-text file placed at the root of a domain (e.g., `example.com/robots.txt`) that provides directives to search engine crawlers about which parts of the site they are allowed or disallowed from crawling. It uses the Robots Exclusion Protocol, with `User-agent` to target specific bots and `Disallow` or `Allow` directives to control access to paths.
Robots.txt controls crawl access, not indexing — a URL blocked by robots.txt can still appear in search results if other sites link to it, though its content cannot be crawled. For preventing indexing, the noindex meta tag is the correct tool. Common uses for robots.txt include blocking crawlers from staging environments, admin sections, duplicate parameter-based URLs, and internal search results.
Why it matters for SEO
Robots.txt is the first line of defense for controlling how search engines interact with a site. Misconfigured robots.txt files are among the most common causes of catastrophic SEO issues — a single mistaken Disallow directive can block an entire site from being crawled and indexed.
Free tools to help with Robots.txt
Ready to put Robots.txt into practice?
LazySEO automates keyword research, content writing, and publishing — so you rank without the manual work.
Try LazySEO for $1