Robots.txt Tester
Enter any domain to read its robots.txt. The tool shows which search engines and AI crawlers are allowed, lists every rule per user agent, finds the sitemap directives and flags anything blocked by accident.
Short answer
robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may request. It controls crawling, not indexing: a URL blocked here can still appear in search results, listed without a description, because Google was never allowed to look at it. To keep a page out of the index you need a noindex tag, and Google has to be able to crawl the page to see it. The file is also public, so it is not a security control. What is newly worth checking is AI crawler access, since GPTBot, ClaudeBot, PerplexityBot and Google-Extended are all governed from here.
What robots.txt controls, and what it does not
Most of the trouble with robots.txt comes from expecting it to do jobs it was never designed for. It answers exactly one question: may this crawler request this path.
| Goal | Is robots.txt the tool? | What to use instead |
|---|---|---|
| Stop a crawler requesting a path | Yes, this is the job | Disallow under the relevant user agent |
| Keep a page out of search results | No | A noindex meta tag or X-Robots-Tag, on a page that stays crawlable |
| Keep content private | No, and it advertises the path | Authentication. The file is public |
| Stop AI training on your content | Yes, for crawlers that honour it | Disallow GPTBot, ClaudeBot, Google-Extended and similar |
| Stay visible in AI search answers | Yes, by not blocking the search bots | Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Reduce Googlebot's crawl rate | No, Crawl-delay is ignored | The crawl rate settings in Search Console |
| Tell crawlers where the sitemap is | Yes | A Sitemap directive with an absolute URL |
The contradiction that costs people pages
Blocking a URL in robots.txt and adding a noindex tag to it at the same time. Google cannot read the noindex, because you have told it not to fetch the page. The URL stays in the index as a bare link indefinitely.
AI crawlers, and the choice most sites have not made
The AI companies split their crawlers in two: one that gathers content for training models, and one that fetches pages to answer a user's question right now. Blocking the first is a licensing decision. Blocking the second removes you from the answers.
| User agent | Operator | What it is for |
|---|---|---|
| GPTBot | OpenAI | Collecting content for model training |
| OAI-SearchBot | OpenAI | Fetching pages for ChatGPT search results |
| ClaudeBot | Anthropic | Collecting content for model training |
| Claude-SearchBot | Anthropic | Fetching pages for Claude search results |
| Google-Extended | Gemini training and grounding, separate from Googlebot | |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Applebot-Extended | Apple | Opting out of Apple Intelligence training |
| Bytespider | ByteDance | Collecting content for model training |
Note that Google-Extended controls Gemini only. Blocking it does not affect Googlebot or your position in Google Search, which is the question most people are actually asking when they consider it.
How to audit a robots.txt file
Six checks in order. The third one catches the most common misreading of the whole format.
- 01
Fetch the file and confirm it exists
robots.txt lives at the root of the host and nowhere else: example.com/robots.txt. A subdirectory copy is ignored, and each subdomain needs its own. A 404 here means every crawler assumes full access, which is fine but is a decision you should make deliberately.
- 02
Read the wildcard block first
The User-agent: * block is what applies to any crawler with no rules of its own, which is most of them. Anything overly broad here, such as a bare Disallow: /, blocks the entire site for everything you have not named explicitly.
- 03
Check the search engines you care about
Googlebot, Bingbot and the rest each read the most specific block that names them, and ignore the wildcard block entirely when one exists. A rule you added to the wildcard block does not apply to Googlebot if Googlebot has its own section.
- 04
Decide, explicitly, about AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest are governed from this file. Plenty of sites block them without having chosen to, because a plugin or a template shipped the rules. Decide separately for training crawlers and for search crawlers.
- 05
Confirm the Sitemap directive is present and correct
A Sitemap line gives every crawler an absolute URL for your sitemap. Google can also learn it from Search Console, but other crawlers rely on this line. It must be an absolute URL and it sits outside any user-agent block.
- 06
Check nothing important is blocked by accident
Blocking a directory of CSS or JavaScript stops Google rendering your pages properly. Blocking a path you also want deindexed is worse than useless, because Google has to crawl the page to see the noindex tag.
Five robots.txt mistakes worth checking for
1. A staging Disallow: / that shipped
The site is built behind a blanket block, the launch checklist misses it, and nothing gets crawled. It is the robots.txt equivalent of a leftover noindex, and it is usually found weeks later by someone wondering why the traffic never arrived.
2. Assuming the wildcard block applies to everyone
It does not. A crawler with its own named block ignores User-agent: * entirely. Adding a rule to the wildcard block while Googlebot has a section of its own changes nothing for Googlebot.
3. Blocking CSS and JavaScript directories
Google renders pages to see them the way a visitor does. A blocked stylesheet or script directory means it renders a broken version of your page and judges what it sees. This was common advice a decade ago and has been wrong for most of that time.
4. Relying on it to hide things
Every robots.txt is public at a predictable URL, and scrapers read them specifically to find interesting paths. Listing a directory here is closer to publishing a map than to locking a door.
5. Letting the file 5xx
Google treats a persistent server error on robots.txt as a site-wide disallow. A missing file grants full access; an unreachable one denies it. Monitor availability, not just contents.
Then check the sitemap it points at
A Sitemap directive is only useful if the file behind it is valid. The sitemap validator checks it against the sitemaps.org protocol, and the sitemap finder locates one when robots.txt does not declare it.
What this tool checks
AI crawler access
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended
Search engine access
Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider and Slurp
Sitemap directives
Every Sitemap line declared in the file
Rule analysis
All Allow, Disallow and Crawl-delay rules, grouped by user agent
Blocking issues
Bots blocked in ways that look unintentional
Raw file view
The complete file, because sometimes you just need to read it
A missing file is safe. An unreachable one is not
Frequently asked questions
What is robots.txt?
A plain text file at the root of a domain that tells automated crawlers which paths they may request. It is usually the first thing a crawler fetches, before it looks at any page. Rules are grouped by User-agent, with Allow and Disallow directives under each, and the file is public: anyone can read yours.
Does robots.txt stop a page being indexed?
No. It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results if other pages link to it, listed without a description because Google was never allowed to look at it. To keep a page out of the index you need a noindex tag, and Google has to be allowed to crawl the page in order to see that tag.
So how do I actually remove a page from search?
Add a noindex robots meta tag or an X-Robots-Tag header, and leave the page crawlable so Google can find the directive. Once the page has dropped out of the index you can block it in robots.txt if you also want to stop the crawling. Doing both at once is the mistake: the block hides the very instruction you need Google to read.
Is robots.txt a security control?
No, and treating it as one is actively harmful. The file is public, so listing /admin or /internal-reports in it advertises those paths to exactly the people you were hoping to keep away. Well-behaved crawlers obey it; nothing else has to. Anything that must stay private needs authentication.
Does robots.txt affect AI search visibility?
Yes. The major AI crawlers read and respect robots.txt, so blocking them removes your content from the systems they feed. A lot of sites are currently blocking them without having decided to, having inherited a rule set from a plugin, a template or a security default rather than making a call.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for OpenAI's model training. OAI-SearchBot crawls for ChatGPT's search results. They are separate user agents, so you can block training while allowing search. Anthropic follows the same split with ClaudeBot for training and Claude-SearchBot for search, and Google separates Google-Extended from Googlebot for the same reason.
Should I block AI crawlers?
It depends on what you sell. If your content is the product, blocking training crawlers is a defensible choice. If you want to be cited in AI answers, blocking the search crawlers costs you that visibility entirely. The common middle position is to block training and allow search, which the separate user agents exist to make possible.
How do the rules resolve when they conflict?
A crawler uses the most specific User-agent block that names it, and ignores every other block including the wildcard. Within that block, Google resolves Allow against Disallow by the length of the matching path, with the longer match winning, and treats a tie as Allow. This is why an Allow for a subpath can override a broader Disallow.
Do Allow and Disallow support wildcards?
Google and Bing support * for any sequence of characters and $ to anchor the end of a URL. So Disallow: /*?* blocks anything with a query string, and Disallow: /*.pdf$ blocks PDFs. These are extensions to the original standard, so less common crawlers may not honour them.
What happens if robots.txt returns a server error?
Google treats a persistent 5xx as though the whole site were disallowed, and will stop crawling. That makes an unreachable robots.txt considerably more dangerous than a missing one: a 404 means full access, while a 503 means no access. Monitor the file's availability, not just its contents.
Should I have a Sitemap directive in robots.txt?
Yes. One line, an absolute URL, outside any user-agent block: Sitemap: https://example.com/sitemap.xml. Google can also learn about your sitemap from Search Console, but other search engines and most AI crawlers rely on this directive to find it at all. You can list several.
Does Crawl-delay work?
Not for Google, which ignores it entirely and asks you to manage crawl rate in Search Console instead. Bing and Yandex do honour it. It is harmless to include for the crawlers that read it, but it is not a way to slow Googlebot down.
Does every subdomain need its own robots.txt?
Yes. The file applies to the exact host, protocol and port that served it, so blog.example.com and example.com each need their own, and the https version does not cover the http one. This surprises people during migrations, when a new subdomain launches with no file and inherits nothing.
Other crawl and indexing tools
Website URL Extractor
Crawl any website and download every URL it links to as CSV, TXT, or JSON. Works straight from the live site, no sitemap required.
Use toolRedirect Chain Checker
Trace every redirect hop with status codes and response times. Detect loops, long chains, protocol downgrades, and 302 misuse.
Use toolPage Indexability Checker
Find out why a page is not indexed. Checks status codes, noindex tags and X-Robots-Tag headers, robots.txt rules, canonicals, sitemap membership, and rendered content.
Use toolBroken Link Checker
Find broken internal and external links on any page. Checks up to 50 links for 404 errors, redirects, and timeouts.
Use toolRobots.txt Generator
Build a correct robots.txt from presets for WordPress, ecommerce, and AI crawlers. Decide which bots to allow, add your sitemap, copy the file.
Use toolAudit crawler access across your entire site
robots.txt is one file, and crawl access is decided by that file plus every meta robots tag, X-Robots-Tag header and canonical on the site. CrawlRaven checks all of them together and tells you which pages are actually reachable.