A robots.txt is four directives and a handful of conventions, and still most files on the web contain at least one line that does nothing. This generator assembles the file from options: a preset as the base, AI crawlers from a list that says what each bot is and when we last checked it, per-bot rules where you need them, and the Sitemap line. The result is re-parsed live through the same RFC 9309 matcher our tester uses, so what you download is a file that verifiably parses into the groups and rules you meant.
What the generator builds
The left side is the policy, the right side is the file, always in sync. The presets cover the three files that make up most of the web's robots.txt population: allow everything (one * group with an empty Disallow), block everything (Disallow: /, the staging-site standard), and block AI training while leaving search alone. From any preset you refine: toggle individual AI crawlers, set the default for unnamed bots, give single crawlers their own Allow and Disallow paths, add the sitemap URL.
The generated file keeps a deliberate structure: per-bot groups first, then the AI block as one group with stacked User-agent lines, then the * group, then the Sitemap line. Order does not matter to parsers, since group choice and match length decide everything, but it matters to the next person who opens the file, and robots.txt files get read by humans more often than most config files.
What belongs in a robots.txt
Start from empty, not from a template. The useful content of a typical robots.txt is short.
- Crawl-waste endpoints. Internal search results (
/search,/*?q=), faceted and sorted listing variants (/*?filter=,/*?sort=), cart and checkout, print views. These generate unlimited URL variants with no search value, and on large sites they eat the crawl budget that should go to real pages. - The Sitemap line. One line, no downside.
- Possibly nothing else. A small site with clean URLs needs exactly the allow-all skeleton plus the sitemap.
What does not belong there, despite appearing in every copied template: CSS and JavaScript paths (Google renders pages and needs them), image directories you want in image search, and anything confidential. The file is public and machine-readable, and a Disallow: /internal-reports/ line is an invitation with a map attached. Access control is authentication's job; robots.txt only asks polite crawlers to look away.
One structural rule is worth internalising because the generator quietly relies on it: a crawler follows only its most specific matching group. If you give Bingbot its own group with one rule, Bingbot ignores the whole * section. Per-bot groups therefore need to be complete policies, not additions, and the generator's per-bot editor treats them that way.
The AI-crawler list, dated
Any list of AI crawlers is a snapshot, and undated lists rot silently: tokens get added, renamed and retired every few months. Ours is dated early 2026, and this is what each entry is.
| Token | Operator | What it does |
|---|---|---|
GPTBot | OpenAI | Crawls for model training |
ChatGPT-User | OpenAI | Fetches a page on demand when a ChatGPT user asks; per OpenAI not used for training |
Google-Extended | Control token for Gemini training, read by Googlebot; blocks training use, not search | |
ClaudeBot | Anthropic | Crawls for model training |
CCBot | Common Crawl | Builds the public archive many training sets start from |
PerplexityBot | Perplexity | Builds their answer-engine index |
Bytespider | ByteDance | Training crawler; repeatedly reported to ignore robots.txt |
Amazonbot | Amazon | Feeds Alexa and Amazon AI products |
Meta-ExternalAgent | Meta | Crawls for AI training |
Applebot-Extended | Apple | Control token for AI training, read by Applebot; Siri and Spotlight stay unaffected |
The "block AI training" preset deliberately selects only the training side: GPTBot, Google-Extended, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent and Applebot-Extended. It leaves ChatGPT-User and PerplexityBot alone, because those fetch pages when a human asks a question about your site, which is closer to a visit than to dataset building. Whether you want that traffic is a judgement call, which is why they are separate toggles rather than part of the preset.
Two entries are not crawlers at all. Google-Extended and Applebot-Extended never appear in your logs; they are flags that Googlebot and Applebot check before content goes into AI training. Blocking them is the rare robots.txt change with zero crawl-side effect.
A request, not a firewall
robots.txt has no enforcement mechanism. It is a text file that well-behaved crawlers fetch and choose to honour, and the operators in the table above document that they do. That is worth something real: the major training pipelines respect it, and opting out via GPTBot or Google-Extended is the supported, documented path.
It is also worth being precise about what it cannot do. It does not remove content from datasets collected before the block existed. It does not bind a scraper that identifies as Chrome. It does not slow down a crawler that never reads the file. When server logs show a blocked bot still fetching, the escalation path is technical: filter the user agent or its published IP ranges at the CDN or web server, rate-limit, or use a bot-management layer. We would rather say that plainly than sell a text file as protection; the file states policy, the server enforces it.
Allow inside a blocked directory
The pattern the per-bot editor exists for: block a directory, reopen one thing inside it. Match length decides every robots.txt conflict, so the reopening rule just has to be longer than the blocking one.
| Goal | Rules |
|---|---|
| Block wp-admin, keep AJAX working | Disallow: /wp-admin/ + Allow: /wp-admin/admin-ajax.php |
| Block /docs/, publish one branch | Disallow: /docs/ + Allow: /docs/public/ |
| Block parameters, keep pagination | Disallow: /*? + Allow: /*?page= |
Order plays no role, only octet counts, and a tie between an Allow and a Disallow of equal length goes to the Allow. The full matching mechanics, including how * and $ count toward match length, live on the robots.txt tester page, where you can also paste the generated file and watch which line wins for any URL and bot.
The Sitemap line
The Sitemap directive is the discovery half of robots.txt: it tells any crawler where your XML sitemap lives without waiting for you to submit it to each engine's console. It stands outside the group structure, so it applies regardless of which User-agent section it sits near, and several lines are allowed for split sitemaps.
The URL must be absolute, including scheme and host. That requirement trips people up because everything else in robots.txt is a relative path; Sitemap: /sitemap.xml is ignored. The generator validates this as you type. Pointing the line at a sitemap index file is the tidy setup for larger sites: robots.txt carries one line, the index carries the rest. Building the sitemap itself is a separate job with its own rules: 50,000 URLs and 50 MB uncompressed per file, and a lastmod that Google ignores entirely once it catches you stamping it on every deploy.
Location, name and encoding
Crawlers fetch exactly one URL: /robots.txt at the root of the host, lowercase, that scheme and that subdomain. Google's robots.txt introduction spells out the scoping: shop.example.com has its own file, example.com/blog/robots.txt governs nothing, and a site living under a path it does not own the root of cannot use robots.txt at all.
The file itself: UTF-8 plain text without a BOM, served as text/plain with a 200 status. Each of those has a failure mode we have actually seen. A BOM makes strict parsers drop the first line and with it the first group header. A text/html content type usually means a framework caught the route and served an error page with status 200, which some crawlers then try to parse as rules. And a 5xx on /robots.txt is worse than a 404: a missing file means "crawl everything", a server error makes Google stop crawling entirely for the first hours and fall back to a cached copy after that.
Generator vs writing it by hand
A robots.txt is short enough to type, and plenty of good ones are typed. The reasons to assemble it from options anyway are the same reasons config generators exist at all: the syntax is easy, the semantics are not, and the file gives no feedback when you get the semantics wrong. A hand-written file with Disallow: admin/ or a group header lost in an edit works exactly like a correct file, minus the part you meant.
The other reason is drift. Most hand-maintained robots.txt files started life as a copied template, and templates age badly: the Noindex: lines kept from a 2018 blog post, the blocked /css/ directory from the pre-rendering era, AI crawler names from a list that was current two years ago. Rebuilding the file from a maintained, dated option set is a cheap way to shed that sediment. A CMS complicates things further: WordPress, for instance, answers /robots.txt with a virtual file unless a physical one exists, and several SEO plugins inject their own lines, so the file you edit is not always the file crawlers see. After any deploy, fetching https://yoursite/robots.txt and comparing it against what you intended is the check that catches this class of surprise.
What no generator replaces is the decision about what to block, and the check that the file does what you decided. The first is your call; for the second, the finished file pastes straight into the tester, which is the other half of this pair.
Why the output checks itself
The note under the generated file is not decoration. Every change re-parses the output through the same RFC 9309 parser and matcher that powers our tester: the groups and rules are counted from the parsed result rather than from the form state, and two spot verdicts are computed live, one for the first blocked AI crawler against / and one for the anonymous * crawler. "3 groups, 5 rules, parsed clean" is a statement about the actual bytes you are about to deploy.
The check also catches the one genuinely confusing interaction this tool can produce: naming the same crawler in the AI block and in a per-bot group. RFC 9309 merges groups that share a token into a single rule set, so a GPTBot per-bot group with an Allow does not override the AI block, it joins it, and the merged rules fight it out by match length. The self-check points that out the moment it happens instead of letting the file ship with a surprise in it. For anything beyond the spot checks, the finished file pastes straight into the tester, URLs and bot matrix included; generating and verifying with the same matcher is the whole point of the pair.
robots.txt questions
What should a robots.txt file contain for a normal website?
Less than most examples suggest. A typical site needs one * group that disallows the endpoints crawlers waste time on (internal search results like /search?q=, cart and checkout pages, infinite parameter combinations), a Sitemap line with the absolute sitemap URL, and nothing else. Login and admin paths are commonly listed too, though listing them also advertises them; they should be protected by authentication either way, not by robots.txt. Do not block CSS, JavaScript or your API routes that render content, and resist copying long boilerplate blocklists from templates: every line is one more chance to block something you wanted crawled.
How do I block GPTBot and other AI crawlers in robots.txt?
Give each crawler token its own User-agent line and one shared Disallow: /. Consecutive User-agent lines share the rules that follow, so "User-agent: GPTBot", "User-agent: ClaudeBot", "User-agent: CCBot" followed by a single "Disallow: /" blocks all three in four lines. The tokens as of early 2026: GPTBot (OpenAI training), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, Meta-ExternalAgent, plus the control tokens Google-Extended and Applebot-Extended, which are read by Googlebot and Applebot rather than crawling themselves. Blocking these does not affect normal search crawling as long as Googlebot and Bingbot keep their access.
Does blocking AI crawlers in robots.txt actually work?
For the named companies, yes, as far as anyone can verify: OpenAI, Anthropic, Google and Common Crawl document that their crawlers honour robots.txt, and server logs generally back that up. But robots.txt is a voluntary protocol. It does not bind crawlers that never announce themselves, scrapers that fake a browser user agent, or datasets already collected before you added the block. Bytespider in particular has been reported repeatedly to ignore the file. Treat robots.txt as the policy statement that compliant crawlers follow, and enforce against the rest at the server or CDN level (user-agent and IP filtering, rate limiting, bot-management products) if it matters to you.
What is the difference between Google-Extended and GPTBot?
GPTBot is a real crawler with its own fetches: block it and OpenAI stops downloading your pages for training. Google-Extended never fetches anything. It is a control token that Googlebot checks before your content is used for Gemini and other Google AI training; the crawling itself continues as ordinary Googlebot traffic for search. So blocking Google-Extended opts you out of Google AI training without touching your search presence, while blocking Googlebot would take you out of both. Applebot-Extended works the same way relative to Applebot.
How do I allow one file inside a disallowed directory?
Add an Allow rule whose pattern is longer than the Disallow that covers it. The classic WordPress example: Disallow: /wp-admin/ together with Allow: /wp-admin/admin-ajax.php. The Allow pattern has more octets, so it wins the longest-match comparison for that one file while the rest of the directory stays blocked. This works for whole subtrees too: Disallow: /docs/ plus Allow: /docs/public/ reopens one branch. What does not work is relying on order; robots.txt has no first-match rule, only match length decides, so put the pair wherever it reads best.
Where does the Sitemap line go, and can I list several sitemaps?
Anywhere in the file, and yes. Sitemap is an independent directive, not part of any User-agent group; crawlers collect every Sitemap line regardless of position, though by convention it sits at the top or bottom. Multiple lines are fine and normal for sites with separate sitemaps for pages, images or news. Two requirements: the URL must be absolute including scheme and host (https://example.com/sitemap.xml, not /sitemap.xml), and it may also point at a sitemap index file that references the others, which keeps robots.txt to a single line.
Where must the robots.txt file be located?
At the root of the exact host and scheme it should govern: https://example.com/robots.txt. Crawlers request that one path and nothing else; a robots.txt in a subdirectory has no effect. Every subdomain needs its own file (shop.example.com does not read example.com's), and technically every scheme and port combination is its own scope, though in practice serving the same file over http and https is what everyone does via redirect. If your site lives under a path on someone else's domain, like username.github.io/project/, you cannot control robots.txt at all, because it would have to live at the domain root.
What encoding and format does robots.txt need?
Plain text, UTF-8, one directive per line, and ideally no BOM: RFC 9309 tells parsers to tolerate a byte order mark, but not every implementation does, and a swallowed first line orphans the rules under it. Lines are name-colon-value; # starts a comment that runs to the end of the line. Serve it with Content-Type: text/plain, not text/html; some crawlers refuse to parse an HTML error page that answers 200 on /robots.txt, and that failure mode is invisible until you look. Line endings do not matter, both LF and CRLF parse everywhere.
Should I block CSS and JavaScript files in robots.txt?
No. Google renders pages before indexing them, and a blocked stylesheet or script means rendering with missing pieces: layout shifts it cannot resolve, content that never appears because the script that loads it was disallowed. Google has warned since 2014 that blocking rendering resources can hurt rankings, and the "blocked resources" section of the URL inspection tool exists to show exactly this. The old advice to block /js/ and /css/ predates rendering crawlers and survives only through copied template files. The same goes for API endpoints your frontend fetches content from.
Do I need a robots.txt file at all?
No. A missing robots.txt returns a 404 and crawlers read that as "no restrictions", which is a perfectly healthy state for a small site where everything may be crawled. Reasons to add one anyway: the Sitemap line gives crawlers a discovery hint that costs nothing, blocking crawl-waste endpoints (internal search, faceted URLs) saves crawl budget on larger sites, and an explicit allow-all file stops the daily 404 noise for /robots.txt in your logs. What a missing file does not do is hide anything, since no file means everything is allowed. The empty-file case behaves the same: no rules, no restrictions.