robots.txt Tester
Test any path on any site against your robots.txt — for Googlebot, Bingbot, GPTBot, or any custom user-agent. Follows Google's official longest-match-wins rule so you see exactly which line allowed or blocked the crawl.
1 · Load robots.txt
Enter a domain — we'll fetch its
/robots.txt. Or paste your own below.2 · robots.txt content
3 · Test a URL
How rule matching works
• Longest match wins. If both
• On a tie, Allow beats Disallow. This is Google's documented behaviour.
• Wildcards.
• User-agent matching. We pick the most specific UA block — a
• No match = Allowed. Absence of any rule means the crawler is free to crawl.
Allow: /blog/private/draft and Disallow: /blog/ match, the longer Allow wins.• On a tie, Allow beats Disallow. This is Google's documented behaviour.
• Wildcards.
* matches any sequence, $ anchors to end of URL.• User-agent matching. We pick the most specific UA block — a
Googlebot block overrides the * block for Googlebot.• No match = Allowed. Absence of any rule means the crawler is free to crawl.
These tools are free and need no account.
They check one page at a time. RankCause itself does the thing they can’t:
it watches your pages over time and tests whether a change you made
actually moved a ranking.