Shopify robots.txt: The Default Is Fine Until You Have Tag Pages

The short answer: leave it alone unless one of six cases applies

Almost every Shopify robots.txt problem I see in a crawl started with someone editing a file that was already right. The nine stores I fetched on September 25, 2026 all serve a generated file that blocks cart, checkout, account, sorted views and multi-filter URLs. The ones in trouble had a hand-written template.

So the answer to "should I edit my Shopify robots.txt" is no, unless you can name the URL type you are blocking. Six qualify: single-tag pages, vendor and type collections, single-filter URLs you never want indexed, internal search on the current default, a second market domain, and AI crawlers you have decided to refuse.

Everything below is quoted from live files and Shopify's own documentation. That matters, because Shopify quietly runs two versions of the default, and the one your store prints depends on whether a theme template exists.

What Shopify's default robots.txt contains today

I fetched /robots.txt from tentree.com, colourpop.com, brooklinen.com, mvmt.com, deathwishcoffee.com, allbirds.com, gymshark.com, kyliecosmetics.com and rothys.com. Five of them serve the same file once you strip the domain and shop ID. That is the current default, and its User-agent: * group runs like this:

• An explicit Allow: /, then 22 Allow lines that rescue product, collection, page and blog handles containing the words account, orders or checkout (for example Allow: /products/checkout).
• Private and transactional: Disallow: /admin, /cart/, /checkout, /checkouts/, /orders and /account, with Allow: /account/login as the exception.
• Shopify-internal endpoints: Disallow: /services, /sf_* and /cdn/wpm/*.js (the web pixels script).
• AJAX surfaces: Disallow: /cart.js and /recommendations/products.
• Filter, sort and preview traps: Disallow: /collections/*sort_by*, /collections/*+*, /collections/*filter*&*filter*, /blogs/*+*, and the ls, oseid, preview_theme_id and preview_script_id parameters.
• One more group and a sitemap: a User-agent: adsbot-google group repeating the checkout and internal rules, then Sitemap: https://yourstore.com/sitemap.xml.

That is 23 Allow and 40 Disallow lines in the main group, two groups in total. The file also opens with a comment block for AI agents (agents.md, UCP endpoints), which you never need to edit.

Two default files are live: current and legacy

The other four stores print the older rule set: no Allow lines, a first line of Disallow: /a/downloads/-/*, plus Disallow: /search, /policies/, /apple-app-site-association and /.well-known/shopify/monorail. On rothys.com, the cleanest example, that is 44 Disallow lines, followed by separate groups for Nutch, AhrefsBot and AhrefsSiteAudit.

Shopify's Help Center page on editing robots.txt.liquid explains the split: a template that outputs the default rule groups "can include rules that the default file no longer has, such as Disallow: /search and Disallow: /policies/", and deleting the template restores the current default.

The one-line test: if your file contains Disallow: /search, you have a template. On the four legacy stores, that search block is the only legacy rule worth keeping.

Why the default is right for most stores

A robots.txt file has one job: stop crawlers spending requests on URLs that will never rank. Google's introduction to robots.txt is explicit that it is "not a mechanism for keeping a web page out of Google". On a store, the URLs worth blocking fall into three buckets, and the default covers all three:

• Per-customer pages. Cart, checkout, orders, account. Blocked in every version of the file.
• Infinite combinations. Sort orders, stacked filters, stacked tags. Google's faceted navigation guidance recommends robots.txt for exactly this, and the default blocks the combinations that explode. The platform-agnostic order for facets, pagination and crawl budget is in technical SEO for ecommerce.
• Platform plumbing. AJAX endpoints, preview parameters, the pixels script. Shopify blocks these before you would think to.

It is also maintained: the current file dropped /policies/ and gained the Allow exceptions without anyone touching a theme. And a store with a few hundred products is nowhere near a crawl budget problem; our SaaS SEO vs ecommerce SEO comparison covers when facets start to matter.

The six cases that justify a robots.txt.liquid template

Each case names a URL type the default leaves crawlable, gives the rule, and says when noindex is the better tool. Google's noindex documentation sets the constraint: a page blocked by robots.txt can never show Google its noindex tag, so the two do not combine.

Tag pages under /collections/*/tag-handle

Shopify's tag filtering documentation describes the URL: append /tag-handle to a collection, so /collections/frontpage/new, and stack tags with a plus sign, so /collections/frontpage/new+sale. The default blocks the stacked form only, because Disallow: /collections/*+* needs a plus in the path. A single-tag page has none: colourpop.com/collections/all/vegan returned a crawlable 200 when I checked.

With 30 tags and 20 collections that is up to 600 near-duplicate listings, each carrying the collection's copy and a subset of its products. Block by tag handle, and pin the end of the URL:

Disallow: /collections/*/red$
Disallow: /collections/*/red?*

• The $ matters. Without it, /collections/*/red also swallows /collections/shirts/products/red-dress, and those collection-path product URLs must stay crawlable so Google can read their canonical.
• The second line catches paginated tag pages, which carry a query string.

If Search Console already shows tag URLs as indexed, blocking leaves them indexed without a snippet. Add a noindex on current_tags in collection.liquid first (our Shopify SEO checklist shows how), and block once they drop out.

Vendor and type collections

Two Liquid filters generate listing pages you never created. url_for_vendor outputs /collections/vendors?q=Vendor+Name and url_for_type outputs /collections/types?q=health. Many themes link to both from product pages. Neither default blocks them; kyliecosmetics.com has added its own Disallow: */vendors?q=* for that reason.

Disallow: /collections/vendors?q=*
Disallow: /collections/types?q=*

Block these unless a vendor page is doing real work for a brand query; if it is, give it a proper collection with its own handle and copy.

Single-filter URLs the default leaves open

The default blocks /collections/*filter*&*filter*, which needs two filter parameters joined by an ampersand. A URL with one filter, such as /collections/shirts?filter.v.option.color=red, stays crawlable. That is deliberate: a single facet can be a legitimate landing page for "red shirts".

• Keep crawlable: the two or three facets with search demand, ideally with their own copy.
• Block by parameter: price and availability, which never carry intent. Disallow: /collections/*filter.v.price* and Disallow: /collections/*filter.v.availability*.
• Leave alone: sort_by, already blocked.

Internal search on the current default

This is the one rule the legacy file had and the current file dropped. /search?q=anything is an unbounded URL space with no unique content, and one linked query can send Googlebot into thousands of variants. Add it back:

Disallow: /search

Under Google's robots.txt specification, a path matches everything that starts with it, so /search also covers /search?q=shirt.

Staging previews and market domains

Shopify has no staging path on the live domain. Theme previews use preview_theme_id, already blocked, and a development store sits behind a password. The real case is Shopify Markets, where one theme serves several domains. Shopify's customize robots.txt guide shows host-specific rules through request.host:

{%- if request.host == 'example.fr' -%}
{{ 'Disallow: /en/' }}
{%- endif -%}

AI crawlers: GPTBot, ClaudeBot, PerplexityBot

No version of the default names an AI crawler, so every one follows the User-agent: * group and is allowed. Whether to change that is a business decision, and the vendors split their bots so you can refuse training without leaving AI search:

OpenAI's crawler overview says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Anthropic's page separates ClaudeBot from Claude-SearchBot and Claude-User. Perplexity's page says PerplexityBot "is not used to crawl content for AI foundation models". To refuse training only, add groups after the loop:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

What you must never block on Shopify

Three URL families carry the store, and each is one careless wildcard away from a Disallow:

• /products/ and /collections/. The money pages. In an August 2025 Shopify Community thread a merchant was sure robots.txt was blocking products; the reply: "Robots.txt does not have any disallow line items blocking product pages". No default does.
• /cdn/shop/ theme assets. Your theme's CSS and JavaScript are served from /cdn/shop/t/…/assets/ on your own domain; the default blocks only /cdn/wpm/*.js. Google's intro says not to block resources when their absence makes "the page harder for Google's crawler to understand".
• A bare Disallow: /collections/* to catch tags. It catches every collection too, because Google matches on prefix. Write the tag path in full.

How to edit robots.txt.liquid without losing Shopify's defaults

Shopify's robots.txt.liquid template reference allows plain text but says "it's strongly recommended to use the provided Liquid objects whenever possible", because "the default rules are updated regularly". The five steps below keep the loop intact.

Step 1: Create the robots.txt.liquid template

1. In Shopify admin, open Online Store, find the published theme, and click Edit code.
2. Under Templates, click Add a new template, choose robots as the type, and click Create template.
3. Shopify pre-fills it with the default loop. Do not clear it.
4. Remove any earlier workaround first, such as Cloudflare rules rewriting robots.txt. The Help Center says to do this before editing.

Two cautions from the same page: it is an unsupported customisation, and uploading a theme through the admin does not carry robots.txt.liquid across (command-line uploads do).

Step 2: Keep the default groups loop

The generated template is short. Every rule group lives in the robots.default_groups array, and the loop prints the user agent, each rule, and the sitemap:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Only robots, group, rule, user_agent, sitemap and request resolve here; anything else leaves you a blank file.

Step 3: Append rules inside the * group

To add a Disallow to the main group, test the group's user agent inside the loop and print your lines after the rules. This is Shopify's own pattern from the customize guide, with the vendor, type and search rules from above:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /collections/vendors?q=*' }}
    {{ 'Disallow: /collections/types?q=*' }}
    {{ 'Disallow: /search' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Removing a default rule works in reverse: wrap the rule output in an unless on rule.directive and rule.value. Do it sparingly.

Step 4: Add crawler-specific groups after the loop

Bot groups go below the endfor as plain lines. Google's specification adds a detail that catches people out: a crawler follows only the most specific group matching it, and "user agent specific groups and global groups (*) are not combined".

• Blocking a bot entirely is safe: User-agent: GPTBot, Disallow: /, and nothing else applies.
• Giving a bot a partial group silently drops every * rule for it. A User-agent: Bingbot group with one extra Disallow lets Bingbot crawl your cart. Repeat the defaults inside it or do not create it.
• Extra sitemaps can go anywhere: Sitemap: lines are not tied to a group.

Step 5: Test the live file, then a money page

1. Save, then load yourstore.com/robots.txt in a browser. Look for Liquid error text, a blank file, or your rules inside the wrong group.
2. Test one URL per new rule and one product URL against the file with the tester below.
3. Wait a day, since Google caches robots.txt for up to 24 hours, then read Search Console.

How to test a Shopify robots.txt change

Start with the robots.txt report in Search Console, which shows the fetched file, its last crawl and any parse warnings, and lets you request a recrawl. It does not test a URL against the rules, so pair it with three checks:

• Rule-level: paste your store into the free robots.txt tester. It shows which search and AI crawlers are blocked, which sitemaps the file declares, and syntax Google would ignore.
• Page-level: run one product, one collection and one tag URL through the page indexability checker, which reads the robots.txt rule, meta robots tag, X-Robots-Tag header and canonical together. Product and collection must come back crawlable.
• Template-level: if you would rather not hand-write the Liquid, the Shopify robots.txt.liquid generator builds the loop with your tag, vendor, search and AI-crawler choices in the right group.

Then ask what the block was worth. CrawlRaven joins Search Console, GA4 and a 200-point crawl into one ranked plan, and in ecommerce mode it lists every URL Google reports as "Blocked by robots.txt" beside its clicks before the block. A tag page at zero clicks is a win; a collection at 400 is a rollback.

Lifetime licences launched at $49 for 3 sites. Lifetime pricing steps up as licenses sell, so check the pricing page for the current batch. See pricing.

Why pages are still blocked, or still indexed, after the change

Search Console still shows "Blocked by robots.txt" on a URL I unblocked. Google caches robots.txt for up to 24 hours. Confirm the live file, request a recrawl in the robots.txt report, then use URL Inspection. If the file still shows the old rule, your published theme is not the one you edited.

A tag page I blocked is still in the index. Expected. Google's intro says a disallowed URL "can still be indexed if linked to from other sites", and it now cannot see a noindex either. Remove the block, add noindex in the theme, wait for "Excluded by noindex", then block.

"Indexed, though blocked by robots.txt" on my homepage or a collection. Shopify's Help Center says this usually means Google's data is stale rather than the file being wrong. Fetch the file and confirm no Disallow matches the URL. If one does, that rule is the bug.

• The file is blank or shows Liquid text. A typo in the template. Delete the template to restore the default, then re-add your lines one at a time.
• An app rewrote the file. Some Shopify SEO apps install their own robots.txt.liquid. Check the template before assuming Shopify wrote it; the Shopify SEO audit walks through app-side changes worth reversing.

Shopify robots.txt FAQs

The default file

Does Shopify have a robots.txt by default? Yes. Every store serves a generated file at /robots.txt. The current version opens with Allow: / and disallows admin, cart, checkout, orders, account, sorted views, multi-tag and multi-filter collection URLs, theme previews and internal AJAX endpoints. Shopify updates it centrally.

Why does my file still block /search and /policies/? Your theme has a robots.txt.liquid template that prints the older rule groups. Shopify's Help Center names those two rules as ones the current default no longer has, and says deleting the template restores the current file. Save your custom lines first.

Editing robots.txt.liquid

Can you edit robots.txt on Shopify? Yes, through a theme template named robots.txt.liquid. In the code editor, add a new template of type robots. It renders the default groups through a Liquid loop, and you add rules around that loop. Shopify Support does not help with edits.

How do I reset it? Delete the template from the theme code editor. Shopify serves the current generated file again, including any rules it changed while your template was in place. Copy your custom lines somewhere first.

Blocking and indexing

Should I block AI crawlers on Shopify? Block training crawlers (GPTBot, ClaudeBot) if you want; keep OAI-SearchBot, Claude-SearchBot and PerplexityBot open or you leave those assistants' answers. Shopify Catalog syndication is unaffected either way.

Will blocking a URL remove it from Google? No. A blocked URL can still be indexed from links, and it shows without a description. Blocking also hides the page's noindex tag from Google. Noindex first, block after the URLs drop out.

Where robots.txt sits in the Shopify SEO plan

Robots.txt is a ten-minute job on a Shopify store, and for most stores the ten minutes are spent confirming the default and closing the tab. If Google has found your tag pages, vendor links or search box, write the template, keep the loop, and test a product page before you walk away.

• Before this: fix the collection-path duplicate URLs and blank collection copy in the Shopify SEO audit. They cost more than any crawl rule.
• After this: work through the Shopify SEO checklist, where tag pages and app bloat sit next to robots.txt.
• Background: the robots.txt glossary entry covers syntax that applies on every platform.

Sources

• Shopify Help Center: Editing robots.txt.liquid
• Shopify.dev: robots.txt.liquid template, Customize robots.txt, Filter collections by tag, url_for_vendor and url_for_type
• Google Search Central: Introduction to robots.txt, How Google interprets the robots.txt specification, Block Search indexing with noindex, Managing crawling of faceted navigation URLs
• Search Console Help: robots.txt report
• OpenAI: Overview of OpenAI crawlers, Anthropic: crawler and opt-out guide, Perplexity: crawlers
• Shopify Community: Robot.txt file are blocking my product pages (August 2025)
• Live robots.txt files fetched September 25, 2026 from tentree.com, colourpop.com, brooklinen.com, mvmt.com, deathwishcoffee.com, allbirds.com, gymshark.com, kyliecosmetics.com and rothys.com. Keyword volume and difficulty: DataForSEO Labs, United States, September 2026. SERP observations: DataForSEO SERP API, United States desktop, September 25, 2026.