robots.txt Tester

robots.txt is a file where the syntax is trivial and the semantics are not. Which of your rules actually decides a URL depends on match lengths, group choice and two wildcard characters, and nothing in the file tells you the result. Since Google retired the public robots.txt tester, the usual way to find out has been to guess. This page explains the matching rules the tester implements, and the mistakes it flags because we keep finding them in real files.

What the tester shows

Paste a robots.txt into the first field, the URLs you care about into the second, and pick the crawlers to test as in the row above: Googlebot, Bingbot, GPTBot, ClaudeBot, the * fallback that stands for every unnamed bot, or any custom token. Several at once is the point: you get a verdict matrix, one row per URL and bot.

Each verdict carries its justification: the winning line quoted from your file with its line number, the match length in octets that made it win, and the group that was chosen for that crawler. That trace is what the old Google tester showed and its replacements mostly do not, and it is the difference between knowing that a URL is blocked and knowing which of your 40 lines you have to change.

Below the verdicts, the findings list covers the file itself: rules that sit outside any group, directives that no crawler reads, patterns that can never match, encoding damage. Everything runs in this tab; a robots.txt is public anyway, but the URL paths you test against it often are not.

How rule matching works

The rules come from RFC 9309, which in 2022 finally standardised what Google had been doing for years. Three points cover almost everything:

1. All matching rules compete, order is irrelevant. Every Allow and Disallow in the crawler's group is checked against the URL path. It does not matter which line comes first; robots.txt has no first-match semantics.
2. The longest match wins. Among the rules that match, the one whose pattern has the most octets decides. Disallow: /shop/ is 6 octets, Allow: /shop/sale/ is 11, so /shop/sale/boots is allowed no matter where the two lines stand.
3. A tie goes to Allow. When an Allow and a Disallow match with exactly equal length, the less restrictive rule is used.

Matching is against the path plus query string of the URL, never the scheme or hostname. Rules are anchored at the start of the path: Disallow: /admin matches /admin, /admin/ and /administrator, that last one being a classic accident. The trailing slash is meaningful: Disallow: /admin/ no longer matches /administrator, but it also no longer matches /admin without the slash.

An empty Disallow: matches nothing and therefore allows everything, which makes it the canonical allow-all group. And a rule that does not start with / or * can never match, because every path starts with a slash; the tester flags those instead of quietly ignoring them the way crawlers do.

* and $ in patterns

Two characters carry all the pattern power robots.txt has. * matches any sequence of characters, including none. $ at the end of a pattern anchors it to the end of the path. There is no ^, no character classes, no regex.

The $ plus query string interaction deserves its own warning, because it is the one that bites. Disallow: /*.pdf$ reads like "block all PDFs", but a link that arrives as /whitepaper.pdf?utm_source=x sails through: the path being matched includes the query string, and it does not end in .pdf anymore. If the goal is to keep PDFs out regardless of parameters, Disallow: /*.pdf without the anchor is the right rule, at the price of also matching a path that merely contains ".pdf". Wildcard match lengths count the pattern's octets as written, so /*.pdf$ competes with 7.

Which group a crawler follows

Rules live in groups, and a group starts with one or more User-agent lines followed by its rules. A crawler picks the single group whose token matches it most specifically, token length deciding specificity, and follows only that group. Googlebot-Image looks for Googlebot-Image, settles for Googlebot, and only without either falls back to *.

The consequence people trip over: groups do not stack. The moment a bot has its own group, your carefully built * section stops existing for it. We have seen a file with twenty Disallow lines under * and a two-line User-agent: Googlebot group added later "just for the sitemap note", and those two lines were the entirety of what Googlebot obeyed from then on.

Three structural details round out the picture. Consecutive User-agent lines share one rule set, which is how you address five AI crawlers with a single Disallow: /. Two groups naming the same token are merged, per RFC 9309. And rules above the first User-agent line belong to no group at all and are silently dropped, which is why a file that starts with a stray Disallow: /tmp/ blocks nothing; the tester reports that as an error rather than letting it pass.

Percent-encoding and case

Paths are matched byte-wise and case-sensitively. /Admin/ and /admin/ are different strings, and a Disallow for one does not cover the other. Directive names and user-agent tokens, by contrast, are case-insensitive.

Percent-encoding is normalised before matching, so Disallow: /café/ and Disallow: /caf%C3%A9/ mean the same rule, and both match a request for either spelling of the path. The one exception worth knowing: %2F, the encoded slash, is not the same as a literal / in a path, and the comparison keeps them apart. The tester applies the same normalisation to your patterns and your test URLs, so what you see is what a compliant parser computes.

Disallow is not noindex

The most consequential misunderstanding about robots.txt has nothing to do with syntax. Disallow controls crawling: whether a bot may fetch the URL. It says nothing about indexing: whether the URL may appear in search results. Google can and does index URLs it has never fetched, from anchor text and links alone, and shows them as "Indexed, though blocked by robots.txt", usually as a bare URL without a description.

That leads to a trap with a satisfying wrongness to it: blocking a page in robots.txt and putting <meta name="robots" content="noindex"> on it. The noindex would work, but the crawler that would read it is not allowed to fetch the page, so the tag might as well not exist. The page stays in the index, block and all.

The working combinations, by goal:

• Keep a page out of search results: allow crawling and serve noindex (meta tag or X-Robots-Tag header), or return 404/410, or require authentication.
• Save crawl budget on endless parameter URLs: Disallow is the right tool; you do not care if a faceted URL is technically indexable, you care that Googlebot stops fetching a million of them.
• Keep something confidential: neither. robots.txt is a public file that every curious person reads first precisely because it lists what you consider sensitive. Authentication is the only mechanism that withholds content.

Mistakes that hide in real files

The findings section exists because most robots.txt problems produce no error anywhere, just different crawling than intended. The ones we check for:

• Rules before the first User-agent line. They belong to no group and are dropped. Typically the result of deleting a group header during an edit.
• A UTF-8 BOM at the start. Google strips it, but stricter parsers see \ufeffUser-agent as an unknown directive, drop the line and orphan the whole first group. Windows editors add the BOM when saving as "UTF-8"; save without it.
• Noindex: as a directive. Google honoured it unofficially until September 2019 and ended support with an announcement; it has done nothing since. Files still carry it because it kept working in blog posts long after it stopped working in crawlers.
• Rules without a leading slash. Disallow: admin/ matches nothing, ever, because every path starts with /. Compliant parsers do not fix it for you.
• Crawl-delay. Not an error, but worth knowing that Google ignores it and Bing reads it as seconds between requests, so it is a Bing-only throttle.
• Typos in directive names. Dissalow parses as an unknown directive and the line becomes a no-op. Since robots.txt has no error channel, the typo lives forever unless something points at it.
• Encoding damage. A U+FFFD replacement character in a pattern means the file was saved through the wrong encoding, and that rule can never match a real path again.

Testing AI crawlers next to Googlebot

A modern robots.txt tends to serve two policies at once: normal search crawling stays open, AI training crawlers are shut out. Whether that actually holds is a group-choice question, and it is easy to get wrong in both directions. A GPTBot group with Disallow: / does block OpenAI's training crawler, but if your * group is where the real rules live, remember that GPTBot never reads them once it has its own group. Conversely, blocking * and allowing Googlebot by name shuts out every AI crawler and every other legitimate bot in one line.

The matrix view is built for exactly this check: put / and a couple of real URLs in the list, switch on Googlebot, GPTBot, ClaudeBot and *, and read the four verdicts side by side. The bot row includes the tokens that behave differently than their names suggest: Google-Extended is not a crawler but a control token that Googlebot reads for Gemini training, so blocking it changes AI training use without changing search. ChatGPT-User fetches pages on demand when a ChatGPT user asks, a different token and policy question than GPTBot. If building these groups is the task rather than checking them, the robots.txt generator assembles the file from a dated crawler list.

Online tester vs Search Console and libraries

Google's original robots.txt tester was retired from Search Console in December 2023. What remains there is the robots.txt report, which shows fetch status and parse warnings for your verified properties, but no longer answers "is this URL blocked for this bot" interactively, and it only works for sites you own. Bing's tester still exists inside Bing Webmaster Tools, behind a login and for Bingbot's view.

For bulk or CI use, Google open-sourced the exact C++ parser Googlebot uses (the google/robotstxt repository), and it is the reference we validated this tester's behaviour against: longest match, tie to Allow, group merging and the wildcard semantics all follow it. A local tester like this one sits in the gap between those options: no login, no property verification, any bot token including made-up ones, and a trace for every verdict. What it cannot know is what only the server knows, like whether your CDN serves a different robots.txt to crawlers than to you, so for a final check fetch https://yoursite/robots.txt yourself and paste exactly what came back.

When a rule does not match