Robots.txt Checker
Make sure crawlers can reach the pages you want indexed and stay out of areas you don’t. Review rules per user agent and find the sitemaps the file declares.
Run a check
This tool isn’t available yet
Robots.txt Checker is still in development. The form below previews how it will work, but it can’t be submitted and no checks are run.
Results
Results aren’t available yet
Results will appear here once Robots.txt Checker launches.
How to use Robots.txt Checker
- Enter the address of the site you want to check.
- Run the check to fetch and parse the site’s robots.txt.
- Review the rules for each user agent and test important URLs against them.
What it checks
- File availability and HTTP status
- Rules grouped by user agent
- Allow and Disallow directives
- Sitemap declarations
- Syntax issues and unsupported directives
- Whether a specific path is allowed
Frequently asked questions
Does Disallow stop a page being indexed?
Not reliably. Disallow prevents crawling, but a blocked URL can still be indexed if other pages link to it. To keep a page out of results, allow crawling and use a noindex robots meta tag or header.
Where must robots.txt be located?
At the root of the host, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file.