Robots.txt Tester

What robots.txt tells you

robots.txt lists the parts of a site its owner asks crawlers not to visit. Checking it costs one request at the start of a run and keeps your traffic out of the areas most likely to be defended. This tool reads a pasted copy and tells you which rule applies to a given path and crawler. If a path you need is disallowed, ask the site, use its API, or leave it.

How a crawler picks its rules

This tester follows RFC 9309. A crawler obeys one group, not all the groups that mention it: the group whose product token is the longest match for its own name, falling back to User-agent: * only when no named group matches. That is why a file with a Googlebot block and a * block does not apply both to Googlebot, and why a rule added to the wildcard group can stop applying to the one crawler it was meant for. Groups that repeat the same token further down the file are treated as one group.

Allow, Disallow and who wins

Within the chosen group, the most specific rule wins, and in this tester specificity means the number of characters in the path pattern. So Allow: /admin/public/ beats Disallow: /admin/ for a page inside the public folder, while a longer disallow underneath it beats them both. When an Allow and a Disallow match with exactly the same length, the standard resolves the tie in favor of Allow. Two characters do extra work: * stands for any run of characters, and a trailing $ anchors the pattern to the end of the path, which is the difference between Allow: /$ (the home page only) and Allow: / (the entire site). Both count toward the character total, which is how Disallow: /*.htm outranks Allow: /page.

What this tester does not do

• Nothing is fetched. The file is whatever you paste, so it will not tell you that the live robots.txt differs from this one.
• Paths are compared exactly as written. Two spellings of the same path (one percent-encoded, one not) are treated as different paths, which is also how Google's parser behaves.
• A user-agent is matched on its product token, such as Googlebot, not on a full User-Agent header.
• Non-standard directives such as Host, Clean-param and Noindex are listed but never applied; no major crawler treats Noindex in robots.txt as binding.
• HTTP status handling is out of scope: a 5xx or an unreachable robots.txt makes real crawlers behave as if everything were disallowed, and nothing you paste can show that.

robots.txt is not an access control

A disallowed path is still served to anyone who asks for it, and the file itself is a public list of the directories a site considers sensitive. Treat it as a request from the site rather than a lock, and put real authentication in front of anything that needs protecting. On the crawling side, respecting the file does not replace pacing: a crawl that stays inside the allowed paths but sends requests too quickly can still be rate limited. If you are sizing that pace, the proxy calculator turns a request volume and a deadline into a rate you can defend, and our scraping practice guide covers pacing and the rest.