A full URL works too: the host is stripped, the query is kept, the fragment is dropped.
Disallowed
"Disallow: /private/" (line 10) is the longest matching rule in the "Googlebot" group, at 9 characters.
What decided it
- Path tested
- /private/quarterly-report
- Group applied
- User-agent: Googlebot (line 9)
- Deciding rule
- Disallow: /private/ (line 10)
- Crawl-delay
- Not set for this group
- Sitemaps
- https://example.com/sitemap.xml
Every rule that matched, most specific first
| Line | Rule | Characters |
|---|---|---|
| 10 | Disallow: /private/ | 9 |
Read it before you crawl it
robots.txt is a site telling you, in writing, which parts of itself it would rather you left alone. Honoring it is the cheapest form of good behavior available to anyone running a crawler: it costs a request at the start of a run, it keeps your traffic out of the areas most likely to be defended, and it is the first thing anyone checks when a scraper becomes a complaint. This tool exists to make the file easier to obey, not easier to route around. If a path you need is disallowed, the answer is to ask the site, use its API, or leave it.
How a crawler picks its rules
A crawler obeys exactly one group, not all the groups that mention it. It looks for the group whose product token is the longest match for its own name, and only falls back to User-agent: * when no named group matches at all. That is why a file with a Googlebot block and a * block does not apply both to Googlebot, and why a rule you added to the wildcard group can quietly stop applying to the one crawler you meant it for. Groups repeating the same token further down the file are treated as one group, which is a common surprise in files that have been edited by several people over several years.
Allow, Disallow and who wins
Within the chosen group, the most specific rule wins, and specificity means the number of characters in the path pattern. So Allow: /admin/public/ beats Disallow: /admin/ for a page inside the public folder, while a longer disallow underneath it beats them both. When an Allow and a Disallow match with exactly the same length, the standard resolves the tie in favor of Allow. Two characters do extra work: * stands for any run of characters, and a trailing $ anchors the pattern to the end of the path, which is the difference between Allow: /$ (the home page only) and Allow: / (the entire site). Both count toward the character total, which is how Disallow: /*.htm outranks Allow: /page.
What this tester does not do
- Nothing is fetched. The file is whatever you paste, so it will not tell you that the live robots.txt differs from this one.
- Paths are compared exactly as written. Two spellings of the same path (one percent-encoded, one not) are treated as different paths, which is also how Google's parser behaves.
- A user-agent is matched on its product token, such as Googlebot, not on a full User-Agent header.
- Non-standard directives such as Host, Clean-param and Noindex are listed but never applied; no major crawler treats Noindex in robots.txt as binding.
- HTTP status handling is out of scope: a 5xx or an unreachable robots.txt makes real crawlers behave as if everything were disallowed, and nothing you paste can show that.
robots.txt is not an access control
A disallowed path is still served to anyone who asks for it, and the file itself is a public list of the directories a site considers sensitive. Treat it as a request from the site rather than a lock, and put real authentication in front of anything that genuinely needs protecting. On the crawling side, respecting the file does not exempt you from pacing: a crawl that stays inside the allowed paths and still hammers them will be blocked on rate long before anyone reads your user agent. If you are sizing that pace, the proxy calculator turns a request volume and a deadline into a rate you can defend, and our scraping practice guide covers the rest of the etiquette.