Robots.txt Tester
Enter a domain or paste your robots.txt, choose a crawler and add the URLs you want to check. See whether each one is allowed or blocked, and which rule decides it.
Load a robots.txt
Or paste your own file into the editor.
Robots.txt editor
Applying group
Deciding Allow rule
Deciding Disallow rule
Syntax issue
Test URLs
One per line. Start with / or a full http(s):// URL. The domain itself is not checked.
Result
Fetch a domain or paste a robots.txt, then add the URLs you want to test.
—
- Allowed 0
- Blocked 0
- Unable to determine 0
Syntax Checks —
Checks appear once a robots.txt is loaded.
Sitemaps Found —
Sitemap lines in the file are listed here.
Fetching goes through an EvenSurge server request because browsers cannot read other sites’ files directly. Pasted text is checked in your browser and is not uploaded. Matching follows Google’s robots.txt documentation, and other crawlers may read some rules differently.
