Robots.txt Generator

What robots.txt controls, and what it does not

robots.txt answers exactly one question: may this crawler request this path. That is a narrower job than most people assume, and the gap between what the file does and what it is expected to do produces most of the damage done with it.

The big one is indexing. Blocking a URL here stops crawlers fetching it, but Google can still index the URL from links elsewhere, listed as a bare link with no description. Worse, if you want a page deindexed, blocking it is counterproductive: Google has to crawl the page to see the noindex tag, and the block hides the very instruction you need it to read. Crawl control and index control are different levers, and this file only pulls the first.

Three more mistakes worth ruling out before you deploy anything:

• Blocking CSS and JavaScript directories. Google renders pages the way a visitor sees them. Block the stylesheets or scripts and it renders a broken page and judges what it sees. This was standard advice a decade ago and has been wrong for most of that time.
• Shipping a staging Disallow: / to production. One line built to keep a staging site private will stop the whole production site being crawled, and it is usually discovered weeks later by someone asking where the traffic went.
• Listing secret paths. The file is public at a predictable URL, and scrapers read it specifically to find interesting directories. Listing /admin here is closer to publishing a map than locking a door. Anything private needs authentication.

The directives, and what each one does

The format has stayed small. Five directives and two wildcard characters cover everything this generator emits and nearly every file you will meet in the wild.

The Sitemap line is the one directive that helps crawlers find things rather than blocking them. If you do not have a sitemap yet, the sitemap generator builds one, and if you are not sure where your CMS already publishes one, the sitemap finder will locate it.

Search AI bots and training AI bots are different decisions

The AI companies split their crawlers deliberately: one user agent gathers content for model training, another fetches pages to answer a user's question right now. Blocking the first is a licensing decision that costs you nothing in visibility. Blocking the second removes your content from the answers people actually read. The generator labels each of the 8 bots so the two decisions stay separate.

The common middle position, and the one the Block AI training bots preset encodes, is to block GPTBot, ClaudeBot and Google-Extended while leaving the search and user-request bots allowed. You keep every AI answer citation and opt out of training. Plenty of sites currently run the opposite configuration without having chosen it, because a plugin or a security template shipped a blanket AI block. The robots.txt tester will show you which camp your current file is in.

How to create and deploy a robots.txt

Six steps from preset to verified live file. The second one is where most files go wrong in 2026, because it is the decision most sites have never consciously made.

1. 01

   Pick the preset closest to your setup

   Allow everything is the right starting point for most sites. WordPress adds the admin block with the admin-ajax.php exception, ecommerce blocks cart, checkout, account and the internal search and filter URLs. Every preset can be fine-tuned afterwards.

2. 02

   Decide the AI bot policy deliberately

   Training crawlers (GPTBot, ClaudeBot, Google-Extended) and search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate user agents. Blocking training does not remove you from AI answers. Blocking search does. Toggle each of the 8 bots on its own row rather than inheriting someone else's default.

3. 03

   Add the paths that waste crawl budget

   Internal search results, filter and sort URL variants, and cart or checkout flows generate near-infinite URLs with nothing worth indexing on them. One Disallow line per pattern keeps crawlers spending their budget on pages you want ranked.

4. 04

   Add the Sitemap line

   One line with the absolute URL of your XML sitemap, and the tool places it after the user-agent groups. Google can also learn the sitemap from Search Console, but other search engines and AI crawlers rely on this directive to find it.

5. 05

   Upload the file to the root of your domain

   The file must be reachable at exactly /robots.txt, as in example.com/robots.txt. A copy in a subdirectory is ignored by every crawler, and each subdomain needs its own file.

6. 06

   Test the live file

   Fetch your domain in the robots.txt tester at /tools/robots-txt-tester and confirm the directives parse, the right bots show as allowed or blocked, and the sitemap is picked up. A typo in a user-agent name fails silently, so verify rather than assume.

What this tool generates

Frequently asked questions