Back to SEO Glossary
Technical SEO

What Is Robots.txt?

Robots.txt is a text file at the root of a website (example.com/robots.txt) that tells search engine and AI crawlers which pages and directories they are allowed or disallowed from accessing. It uses User-Agent directives to control access per crawler.

Why Robots.txt matters for SEO

In 2026, robots.txt controls access for both search engines (Googlebot, Bingbot) and AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot). Blocking AI search crawlers means your content won't appear in ChatGPT, Claude, or Perplexity results. Many sites accidentally block AI crawlers, losing all AI search visibility.

Pro tip on Robots.txt

Allow search crawlers (Googlebot, Bingbot) and AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). You can optionally block AI training crawlers (GPTBot, ClaudeBot, Google-Extended) without affecting search visibility. Include a Sitemap directive pointing to your XML sitemap.

Related terms

Crawl Budgetrobots.txt helps conserve crawl budget by blocking low-value pagesXML SitemapThe Sitemap directive in robots.txt tells crawlers where to find your sitemapAI CrawlerAI crawlers like GPTBot and ClaudeBot respect robots.txt directives

Learn more

Free Robots.txt Tester How to Rank in ChatGPT & AI Search

Check robots.txt on your site

Robots.txt GeneratorBuild a correct robots.txt from presets for WordPress, ecommerce, and AI crawlers. Decide which bots to allow, add your sitemap, copy the file.Robots.txt TesterAnalyze robots.txt to see which AI crawlers and search engines are blocked, find sitemaps, and identify access issues.Sitemap FinderFind where any website hides its sitemap. Checks robots.txt directives plus nine common paths, including WordPress and gzipped variants.

Not sure which crawlers your robots.txt is blocking? CrawlRaven's Robots.txt Tester shows exactly which search engines and AI crawlers have access to your site.

Try CrawlRaven Free