Shopify robots.txt.liquid Generator
Build the theme template that customises your Shopify store's robots.txt without overwriting Shopify's defaults. Block tag pages, vendor and type collections, filter parameters and AI crawlers, then copy the file into templates/robots.txt.liquid. Everything runs in your browser.
Every option below adds to Shopify's defaults. The defaults themselves stay inside the Liquid loop, so they keep updating when Shopify changes them.
Shopify URL patterns
AI crawlers
Checked means blocked: a User-agent group with Disallow: / is added after the loop. Training bots cost nothing to block. Search bots fetch pages for AI answers, so blocking one removes your store from that assistant.
templates/robots.txt.liquid
# robots.txt.liquid
# Generated with CrawlRaven: https://crawlraven.com/tools/shopify-robots-txt-generator
# Shopify's default rules are rendered by the loop below and stay current automatically.
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.user_agent.value == '*' %}
Disallow: /collections/*/*
Allow: /collections/*/products/*
Disallow: /collections/vendors?q=
Disallow: /collections/types?q=
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
What each added rule does
Disallow: /collections/*/*Blocks single-tag collection URLs such as /collections/frontpage/new and /collections/all/sale. Shopify's defaults only block tag combinations joined with +.Allow: /collections/*/products/*Carves product URLs reached through a collection back out of the tag block. The longer Allow rule wins, so /collections/frontpage/products/hoodie stays crawlable.Disallow: /collections/vendors?q=Blocks the automatic vendor collections, one per brand you sell. Themes link to them with url_for_vendor, and each one duplicates products already in your real collections.Disallow: /collections/types?q=Blocks the automatic product-type collections built by url_for_type. Same duplication as vendor collections, with no unique copy on the page.
Once the theme is published, fetch your store in the robots.txt tester to confirm the rendered file parses and the defaults are still present.
Short answer
Shopify has no upload for robots.txt. The file at /robots.txt is rendered from templates/robots.txt.liquid, and Shopify serves its own defaults when the template is absent. This generator writes that template: a loop over robots.default_groups outputs every default group's user agent, rules and sitemap unchanged, and your rules are added inside the * group or after the loop.
How Shopify's robots.txt works, and why you cannot upload one
Every Shopify store answers /robots.txt, but no Shopify store has a robots.txt file. The response is rendered by Liquid from templates/robots.txt.liquid, and when that template does not exist, which is the case for every theme as shipped, Shopify renders its own default rules instead.
That design has a consequence people miss: the defaults are not a starting file you edit. They are a Liquid object, robots.default_groups, that Shopify updates on its own schedule. A template that pastes today's rules as plain text freezes them, and the next default Shopify adds never reaches your store.
The loop this generator writes outputs three things for every default group:
- group.user_agent, the User-agent line, such as * or adsbot-google.
- group.rules, each one a directive and a value, rendered as a line like Disallow: /checkout.
- group.sitemap, the Sitemap line for the group when Shopify provides one, guarded by a blank check.
Your additions go in the two places Shopify's customisation guide opens up: a conditional inside the wildcard group, keyed on group.user_agent.value, and plain text after the loop for named crawlers and extra sitemaps. The result is a normal robots.txt to every crawler, with Shopify still in charge of the defaults.
One line break the docs strip
Shopify's add-a-rule example strips the whitespace on both sides of the added line. On live stores that copied it, the first custom rule is glued onto the end of the last default rule, which makes both lines invalid. This template keeps one line break before your first rule, so every directive starts on its own line.
When to customise it, and when to leave the defaults alone
Shopify's defaults already block cart, checkout, account, admin, internal search, sort_by variants, tag combinations joined with + and theme preview URLs. Most of the classic ecommerce robots.txt advice is done for you before you open the editor.
What the defaults leave open are the URL families a theme generates without asking: single-tag pages, vendor and type collections, and every storefront filter combination. On a store with a few hundred products those can outnumber the real pages ten to one, and they are where crawl budget goes first.
| Rule | What it blocks | When to use it |
|---|---|---|
| Disallow: /collections/*/* | Single-tag URLs like /collections/frontpage/new and /collections/all/sale. Paired with Allow: /collections/*/products/* so products reached through a collection stay open | Your theme links to tags and the tag pages have no unique copy, which is nearly always |
| Disallow: /collections/vendors?q= | The automatic per-brand collections built by url_for_vendor | You sell more than one brand and the theme links vendor names |
| Disallow: /collections/types?q= | The automatic per-type collections built by url_for_type | The theme links product types, or an app generates those URLs |
| Disallow: /*?name= and /*&name= | Any URL carrying that query parameter, first or later in the string | Storefront filters (filter.v.price.gte, filter.v.option.color) and variant deep links multiply URLs per option |
| Disallow: /custom/path | Everything beginning with that path, for all crawlers in the * group | Wholesale logins, blog tag archives, app landing pages you never want crawled |
| unless rule.value == '/policies/' | Skips one of Shopify's default rules while the rest keep rendering | You want a default-blocked page such as shipping policy crawlable, and you have checked the exact value in your live file |
| User-agent: GPTBot + Disallow: / | A training crawler: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | You do not want product copy used for model training. Costs no search or AI answer visibility |
| User-agent: PerplexityBot + Disallow: / | A search crawler: Claude-SearchBot, PerplexityBot | Only if you have decided to leave that assistant's answers entirely. Shoppers asking it for recommendations will not see you |
| Sitemap: https://... | Declares a second sitemap after the default groups | A blog or app sitemap hosted outside Shopify's /sitemap.xml |
Two situations where the right template is no template at all:
- A small catalogue with no tag navigation. Under a few dozen products with the theme's tag links switched off, there is nothing for the extra rules to catch. The defaults are enough.
- You are trying to deindex a page. Blocking it hides the noindex tag Google needs to read. Leave the page crawlable, add noindex, and confirm the state with the page indexability checker.
AI crawlers are two decisions, not one
Blocking GPTBot, ClaudeBot, Google-Extended, CCBot or Bytespider opts your copy out of model training and costs nothing in visibility. Blocking Claude-SearchBot or PerplexityBot removes your products from that assistant's shopping answers. The full reasoning, with what each bot fetches, is in the Shopify robots.txt guide.
How to add robots.txt.liquid in the theme code editor
The template lives in the theme, not in a settings page, so the only way in is the code editor. The click path below is the one Shopify documents for the desktop admin; the mobile app reaches the same editor through Store, then Online Store, then Manage all themes.
- From Shopify admin, go to Online Store, then Themes.
- Find the theme to edit, click the three dots, then Edit code.
- In the left sidebar, right-click the Templates folder and choose New File.
- Name the file robots.txt.liquid and press Enter. It cannot be a JSON template.
- Paste the generated template into the editor and save.
Six steps take you from the live default file to a tested template. The first one matters more than it looks: most of what people try to add is already in the defaults, and reading the live file first keeps the template short.
- 01
Read your live default file first
Open yourstore.com/robots.txt in a browser. Shopify already blocks cart, checkout, account, internal search, sort_by variants and tag combinations, so most stores need two or three additions rather than a rewrite. Note anything you plan to remove, because the removal has to match that line exactly.
- 02
Choose the Shopify URL patterns to block
Turn on tag pages if your theme links to tags, and vendor and type collections if it uses url_for_vendor or url_for_type. List the storefront filter parameters your collections use, such as filter.v.price.gte. Leave internal search alone; the defaults cover it.
- 03
Set the AI crawler policy on purpose
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) can be blocked without losing any visibility. Search crawlers (Claude-SearchBot, PerplexityBot) fetch your pages to answer shopping questions, so blocking one removes your products from that assistant. Decide each row rather than blocking the lot.
- 04
Add an extra sitemap or a default-rule removal only if you need one
Shopify's own sitemap is already output by group.sitemap inside the loop, so the extra line is for a blog or app sitemap hosted elsewhere. A removal wraps the rule output in the unless guard Shopify documents; copy the value from your live file so it matches.
- 05
Create templates/robots.txt.liquid in the theme code editor
In Shopify admin go to Online Store, then Themes. On the theme you want, click the three dots, then Edit code. In the left sidebar right-click the Templates folder, choose New File, name it robots.txt.liquid and press Enter. Paste the generated template and save.
- 06
Publish, then test the rendered file
Fetch /robots.txt again and confirm the default lines are still there, your rules appear once each on their own line, and the Liquid tags are gone. Run the domain through the robots.txt tester, then check one tag URL and one product URL in the page indexability checker.
The template belongs to one theme
robots.txt.liquid is a theme file. Publish a new theme later and the template stays behind with the old one, so the store silently returns to Shopify's defaults. Copy it into the new theme before you publish, and put that check on your Shopify SEO checklist.
How to test the file after you publish
Publishing the theme makes the change live at /robots.txt straight away, and that URL is where the check starts, not the editor. A Liquid typo does not throw an error you will see; it renders a file with a missing group or a glued line, and every crawler reads that.
- Open yourstore.com/robots.txt. The default lines (Disallow: /checkout, /cart, /search) must still be there, your rules must appear once each on their own line, and no Liquid tags may remain.
- Parse it. Run the domain through the robots.txt tester to confirm every directive parses and each AI crawler resolves to the access you intended.
- Test one URL of each kind. A tag URL should now report blocked, and a product URL under /collections/ should still report crawlable.
- Check Search Console. Settings, then Crawling, then the robots.txt report shows when Google last fetched the file and whether it parsed.
Over the following weeks the Pages report tells you whether it worked. "Blocked by robots.txt" should rise by roughly the number of tag and filter URLs, and "Crawled – currently not indexed" should fall as the crawler stops spending its visits there.
Know what a blocked path was earning
A tag page that quietly ranked for a long-tail query stops earning the moment the Disallow lands. CrawlRaven's 200-point audit reads Search Console's indexing report against the crawl, so the URLs a rule will block are listed with their clicks first. For a plain file on a non-Shopify site, use the robots.txt generator builds that instead.
What this tool generates
The documented default loop
for group in robots.default_groups, with user_agent, rules and sitemap output exactly as Shopify's docs show
Tag page block
Disallow: /collections/*/* paired with an Allow for /collections/*/products/*
Vendor and type collections
Disallow rules for /collections/vendors?q= and /collections/types?q=
Filter and sort parameters
Two rules per parameter so it is caught first or later in the query string
Markets subfolder mirror
Optional locale-prefixed copy of every path rule, matching how Shopify's own defaults are written
7 AI crawlers, labelled
GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Bytespider, each marked training or search
Default-rule removal
The unless guard from Shopify's guide, one per value, nested rather than chained
Copy or download
The file as robots.txt.liquid, plus a plain-English note for every rule you added
Generate here, verify on the live URL
Frequently asked questions
What is robots.txt.liquid?
robots.txt.liquid is the theme template that renders your Shopify store's /robots.txt. It lives in the theme's templates folder and is not included in any theme by default, because Shopify's generated file suits most stores. Add it only when you need a rule the defaults do not cover, and keep the default loop inside it.
Can I upload a robots.txt file to Shopify?
No. Shopify serves /robots.txt from the robots.txt.liquid template, and there is no file upload for it. To change the rules you create that template in the theme code editor, under Templates. The change belongs to the theme it lives in, so a new theme without the template goes back to Shopify's defaults.
Does this template delete Shopify's default rules?
No. The loop over robots.default_groups outputs every default group, rule and sitemap exactly as Shopify would, and your additions are appended. Shopify recommends this over pasting plain text because it updates the defaults regularly. The one exception is a value you list under Remove a default rule, which the unless guard skips on purpose.
Should I block collection tag pages on Shopify?
Usually yes, if your theme links to tags. Each tag creates a URL like /collections/frontpage/new that repeats products already on the collection page with no unique copy, and 20 tags across 30 collections is 600 such pages. Shopify's defaults only block tag combinations joined with +, so single tags stay crawlable until you add this rule.
What are vendor and type collections?
They are collections Shopify builds automatically at /collections/vendors?q=Brand and /collections/types?q=Type from the vendor and product type fields. Themes link to them with the url_for_vendor and url_for_type filters. Every one duplicates products your real collections already list, so blocking both keeps crawlers on the pages you actually optimise.
Is internal search already blocked on Shopify?
Yes. Shopify's default rules include Disallow: /search, which is why the generator shows that option as already covered instead of adding a second copy. The ?q= parameter on collection URLs is separate, and Shopify's own customisation example uses Disallow: /*?q=* if you want those variants blocked as well.
Will blocking GPTBot remove my store from ChatGPT?
No. GPTBot only gathers pages for OpenAI's model training, so blocking it costs no visibility in ChatGPT answers. ChatGPT's search results come from a separate user agent, OAI-SearchBot, and live user requests from ChatGPT-User. The generator labels each crawler training or search so the two decisions stay separate.
Which AI crawlers should a Shopify store block?
Block the training crawlers if you do not want your product copy used to train models: GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. Leave the search crawlers, Claude-SearchBot and PerplexityBot, allowed, because they fetch your pages to answer shopping questions. Blocking a search crawler removes your products from that assistant's recommendations.
How do I remove one of Shopify's default rules?
Wrap the rule output in an unless guard, which is the method Shopify documents. For each value you list, the generator writes unless rule.directive == 'Disallow' and rule.value == '/policies/' around the rule output. The value has to match the default line exactly, so copy it from your live /robots.txt before relying on it.
Why do added rules need the group.user_agent.value == '*' check?
Because Shopify's default file has more than one group, and a rule added inside the loop without the check would be repeated in every one of them. The check limits your additions to the wildcard group that applies to all crawlers. Named groups such as adsbot-google keep only the rules Shopify gives them.
Does robots.txt stop a Shopify page being indexed?
No. It stops crawling, not indexing. A blocked URL can still appear in Google if other pages link to it, listed without a description. To keep a page out of the index, leave it crawlable and add a noindex tag, which on Shopify usually means a theme.liquid condition or an SEO app rather than a setting.
How long until Google sees the new robots.txt?
Google caches robots.txt and generally refreshes it within about a day, so a change published today is usually honoured tomorrow. Nothing is retroactive: URLs Google already crawled stay in its index until they are recrawled and reprocessed. Fetch your live /robots.txt straight after publishing to confirm the template rendered at all.
Other crawl and indexing tools
Website URL Extractor
Crawl any website and download every URL it links to as CSV, TXT, or JSON. Works straight from the live site, no sitemap required.
Use toolRedirect Chain Checker
Trace every redirect hop with status codes and response times. Detect loops, long chains, protocol downgrades, and 302 misuse.
Use toolPage Indexability Checker
Find out why a page is not indexed. Checks status codes, noindex tags and X-Robots-Tag headers, robots.txt rules, canonicals, sitemap membership, and rendered content.
Use toolBroken Link Checker
Find links that no longer work, internal and external together. Checks up to 50 links on a page for 404 errors, redirects, and timeouts.
Use toolExternal Link Checker
Audit who a page links out to and on what terms. Destinations grouped by domain, with rel nofollow, sponsored and ugc, new-tab safety, and live status.
Use toolInternal Link Analyzer
See which internal links a page makes in its own writing and which come from the template, grouped by destination, with anchor text, nofollow and live status.
Use toolRobots.txt Tester
Analyze robots.txt to see which AI crawlers and search engines are blocked, find sitemaps, and identify access issues.
Use toolRobots.txt Generator
Build a correct robots.txt from presets for WordPress, ecommerce, and AI crawlers. Decide which bots to allow, add your sitemap, copy the file.
Use toolllms.txt Generator
Build a valid llms.txt from your own pages: the H1, blockquote summary and H2 link sections the proposal specifies, checked for relative URLs and duplicates.
Use toolHTTP Header Checker
Read every response header a URL returns, with the redirect chain timed hop by hop and X-Robots-Tag, header canonicals and caching gaps called out.
Use toolShopify App Script Checker
Inventory every script tag on a Shopify product or collection page by origin: theme, Shopify platform, app and third-party. Flags async, defer and blocking scripts, sizes up to 30 external files and lists legacy ScriptTag API scripts.
Use toolRead up on crawling and indexing
A tool tells you what is wrong. These explain what to do about it.
9 Best Screaming Frog Alternatives in 2026 (Free, Cloud & Enterprise)
Read guideTechnical SEO Audit Tools: Every One Finds the Same Issues. They Differ on What Comes Next.
Read guideSitebulb Review 2026: Pricing, Features & Honest Verdict
Read guideTerms this tool checks
- Robots.txt
Robots.txt is a text file at the root of a website (example.com/robots.txt) that tells search engine and AI crawlers which pages and directories they are allowed or disallowed from accessing.
- Crawl Budget
Crawl budget is the number of pages a search engine will crawl on your site within a given timeframe, determined by crawl rate limit (how fast Googlebot can crawl without overloading your server) and crawl demand (how much Google wants to crawl based on popularity and freshness).
- Indexation
Indexation is the process by which search engines add web pages to their searchable database (index).
- Noindex
Noindex is a robots meta tag directive (meta name="robots" content="noindex") that tells search engines not to include a specific page in their search index.
See what a blocked path is costing before you block it
CrawlRaven is Search Console, GA4 and a 200-point crawl joined into one ranked plan. The audit reads Search Console's indexing report against the crawl, so a tag page that was quietly earning clicks shows up before the Disallow removes it. Free plan for 1 site, no credit card.