Robots.txt is a text file at the root of a website (example.com/robots.txt) that tells search engine and AI crawlers which pages and directories they are allowed or disallowed from accessing. It uses User-Agent directives to control access per crawler.
Why Robots.txt matters for SEO
In 2026, robots.txt controls access for both search engines (Googlebot, Bingbot) and AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot). Blocking AI search crawlers means your content won't appear in ChatGPT, Claude, or Perplexity results. Many sites accidentally block AI crawlers, losing all AI search visibility.
Pro tip on Robots.txt
Allow search crawlers (Googlebot, Bingbot) and AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). You can optionally block AI training crawlers (GPTBot, ClaudeBot, Google-Extended) without affecting search visibility. Include a Sitemap directive pointing to your XML sitemap.
Related terms
Learn more
Not sure which crawlers your robots.txt is blocking? CrawlRaven's Robots.txt Tester shows exactly which search engines and AI crawlers have access to your site.
Try CrawlRaven Free