Sitemap URL Extractor

What the extractor handles

Most sitemaps on real sites are not a single flat file. They are an index pointing at several children, split by content type, and often compressed. The extractor deals with all of that and hands back one plain list.

• Sitemap index files. Followed recursively to three levels, across up to 50 child sitemaps.
• Gzipped sitemaps. Both a gzip Content-Encoding header and a raw .xml.gz file are decompressed before parsing.
• Namespace prefixes. Files using a prefix such as sm:urlset parse the same as unprefixed ones.
• CDATA and entities. URLs wrapped in CDATA or carrying escaped ampersands are decoded properly.
• Optional fields. lastmod, changefreq, and priority come through into the CSV alongside each URL.
• Source tracking. Each row records which child sitemap the URL came from, which matters when you are debugging an index.

What you get back

• A searchable, paginated table of every extracted URL.
• CSV with url, lastmod, changefreq, priority, and source sitemap columns.
• Plain text, one URL per line, for pasting into another tool.
• JSON, if you want to script against the output.
• A per-file breakdown showing how many URLs each child sitemap contributed.

What people use a sitemap URL list for

1. 01

   Pre-migration inventory

   Extract every URL before a replatform, so you have a definitive list to build redirects from and to check against afterwards.

2. 02

   Bulk status checking

   Feed the list into a link checker to find which declared URLs are already dead. Our sitemap crawler does this for the first 100 automatically.

3. 03

   Competitor research

   A competitor sitemap tells you their full content inventory, their URL patterns, and, through lastmod, how often they actually publish.

4. 04

   Content auditing

   Group the URL list by path prefix to see where your pages actually live and which sections have grown without anyone deciding they should.

5. 05

   Rank tracker setup

   Import the list rather than adding URLs by hand, so tracking matches what the site actually publishes.

Sitemap list or crawled list: which one you want

Both this tool and the website URL extractor hand you a list of a site's URLs, and they answer different questions. This one reads the sitemap, so it reports what the site says it has. That one crawls the live site by following links, so it reports what the site actually connects to.

• Use the sitemap list when you want speed and completeness. One file, no crawling, and on a well-maintained site it includes pages nothing links to.
• Use a crawl when there is no sitemap, when you suspect the sitemap is stale, or when you need to know how each page is actually reached.
• Use both when the counts disagree, because the gap is the finding. URLs in the sitemap but not the crawl are orphan pages: declared to Google, cut off from your internal links. URLs in the crawl but not the sitemap are pages you link to but never declared.

The sitemap comparison tool does that diff for you once you have both lists.

Reading a sitemap URL list

A flat list of URLs answers a few questions quickly once you sort it.

• Count by path prefix. If one section holds most of your URLs, that is where your crawl budget goes, whether you meant it to or not.
• Look at lastmod spread. Every URL sharing one date means the field is generated, not real, and Google will ignore it.
• Check for parameter URLs. Filter and sort parameters in a sitemap usually mean the generator is indexing faceted navigation.
• Scan for staging hostnames. Migrations regularly leave a handful of staging URLs behind in the generated file.

To see this as a structure rather than a list, the visual sitemap generator groups the same URLs into a navigable tree.

Output formats, and what each one is for

The same list in three shapes. Which one you want depends entirely on what happens to it next.

Three jobs the list does well

A URL list is not interesting on its own. It becomes interesting the moment you compare it against something else: the pages Google has actually indexed, the pages your analytics recorded a visit to, or the same list from before a migration.

• Migration checks. Export before, export after, diff. Anything in the first list and not the second either moved or vanished, and you want to know which before Search Console tells you.
• Coverage gaps. Compare the sitemap against your indexed pages. URLs declared but never indexed are the ones worth investigating.
• Competitor research. A sitemap is public and complete. It is a faster read on what a competitor has invested in than any keyword tool.

The comparison that finds the most is usually the dullest one: your sitemap against your own analytics. A URL declared in the sitemap that has never received a visit is either a page nobody wants or a page nobody can find, and the two problems have completely different fixes.

The most useful thing to do with two exports is to diff them. A JSON export taken before a migration and another taken after tells you exactly which URLs disappeared, which is a question that is otherwise surprisingly hard to answer.

Frequently asked questions