Robots.txt Tester
Paste your robots.txt and test any URL against any crawler to see what is allowed.
Robots.txt Tester Features
Verify your crawl rules before deployment.
Shows exactly which rule matched and whether it is Allow or Disallow.
Test against specific crawlers like Googlebot or the wildcard *.
Displays all applicable rules for the selected user-agent at a glance.
About This Tool
This tool checks a robots.txt file against a specific URL and crawler to tell you whether that page would be allowed or blocked. Because the matching rules combine Allow and Disallow directives with wildcards and the longest-match wins, working out by eye whether a given path is blocked is surprisingly easy to get wrong.
Testing before you publish prevents two costly mistakes: accidentally blocking pages you want indexed, and failing to block ones you meant to. The tester applies the same precedence rules real crawlers use, so the verdict reflects what would actually happen rather than what the file appears to say.
Precedence is where intuition fails: when both an Allow and a Disallow rule match a URL, the more specific (longer) rule wins, not the first one listed. Wildcards and end-of-URL anchors add further subtlety. Testing a real URL against your rules shows the actual verdict and which directive produced it, which is the only reliable way to confirm a nuanced rule set behaves as intended before it goes live.
How to Use It
- Paste your robots.txt content.
- Enter a URL and choose a crawler user-agent.
- See whether that URL is allowed or disallowed, and which rule decided it.
This space is available for sponsored content. Advertise here