Robots.txt Tester
Fetch and test robots.txt rules for any website.
Enter a domain name to fetch and analyze its robots.txt file.
What Is robots.txt?
The robots.txt file is a plain-text file placed at the root of a website (e.g., https://example.com/robots.txt) that tells web crawlers which pages or sections they are allowed or disallowed from accessing. It follows the Robots Exclusion Protocol, a standard that has been in use since 1994. While it is not a security mechanism — crawlers can choose to ignore it — all major search engines like Google, Bing, and Yahoo respect its directives.
How robots.txt Works
The file consists of one or more blocks, each starting with a User-agent directive specifying which crawler the rules apply to. A wildcard (*) targets all crawlers. Each block contains Disallow and Allow rules that use path patterns to control access. A Sitemap directive can also point crawlers to your XML sitemap for better indexing.
Why Test Your robots.txt?
A misconfigured robots.txt can have serious consequences. Accidentally blocking your entire site with Disallow: / will remove it from search results. Overly permissive rules may expose admin pages or private content to crawlers. This tool fetches your robots.txt, parses every rule, and presents them in a clear format so you can verify that your crawl directives are working as intended.
Common Issues
Frequent mistakes include blocking CSS and JavaScript files (which prevents Google from rendering your pages), using incorrect path syntax, forgetting the trailing slash on directories, or having conflicting Allow and Disallow rules. This tool helps you spot these issues by breaking down each rule for every user agent defined in your file.