Robots.txt Tester
Test and validate your robots.txt file. Check which pages search engines can access and identify potential SEO issues.
Test URL Access
Raw robots.txt
Enter a URL and click Get Data to load robots.txt
What is robots.txt?
The robots.txt file is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. It tells search engines which pages they can or cannot access.
Common Directives
User-agent:Specifies which search engine robot to target
Disallow:Tells robots not to access specific pages
Allow:Explicitly allows access to files/directories
Sitemap:Location of your XML sitemap
Crawl-delay:How many seconds between requests
How does this tool work?
- 1
Enter any website URL to fetch and analyze its robots.txt file in real-time
- 2
Test specific paths to see if search engines can access them
- 3
Identify potential SEO issues with blocking rules
- 4
View statistics about rules found in the file
Important Tips
- • Ensure your sitemap URL is listed in robots.txt
- • Don't block CSS or JS files - it may harm rendering
- • Use specific paths rather than blocking entire directories
- • Test your robots.txt with Google's robots.txt tester