Sitemap URL Extractor

What the extractor handles

Most sitemaps on real sites are not a single flat file. They are an index pointing at several children, split by content type, and often compressed. The extractor deals with all of that and hands back one plain list.

• Sitemap index files. Followed recursively to three levels, across up to 50 child sitemaps.
• Gzipped sitemaps. Both a gzip Content-Encoding header and a raw .xml.gz file are decompressed before parsing.
• Namespace prefixes. Files using a prefix such as sm:urlset parse the same as unprefixed ones.
• CDATA and entities. URLs wrapped in CDATA or carrying escaped ampersands are decoded properly.
• Optional fields. lastmod, changefreq, and priority come through into the CSV alongside each URL.
• Source tracking. Each row records which child sitemap the URL came from, which matters when you are debugging an index.

What you get back

• A searchable, paginated table of every extracted URL.
• CSV with url, lastmod, changefreq, priority, and source sitemap columns.
• Plain text, one URL per line, for pasting into another tool.
• JSON, if you want to script against the output.
• A per-file breakdown showing how many URLs each child sitemap contributed.

What people use a sitemap URL list for

1. 01

   Pre-migration inventory

   Extract every URL before a replatform, so you have a definitive list to build redirects from and to check against afterwards.

2. 02

   Bulk status checking

   Feed the list into a link checker to find which declared URLs are already dead. Our sitemap crawler does this for the first 100 automatically.

3. 03

   Competitor research

   A competitor sitemap tells you their full content inventory, their URL patterns, and, through lastmod, how often they actually publish.

4. 04

   Content auditing

   Group the URL list by path prefix to see where your pages actually live and which sections have grown without anyone deciding they should.

5. 05

   Rank tracker setup

   Import the list rather than adding URLs by hand, so tracking matches what the site actually publishes.

Reading a sitemap URL list

A flat list of URLs answers a few questions quickly once you sort it.

• Count by path prefix. If one section holds most of your URLs, that is where your crawl budget goes, whether you meant it to or not.
• Look at lastmod spread. Every URL sharing one date means the field is generated, not real, and Google will ignore it.
• Check for parameter URLs. Filter and sort parameters in a sitemap usually mean the generator is indexing faceted navigation.
• Scan for staging hostnames. Migrations regularly leave a handful of staging URLs behind in the generated file.

To see this as a structure rather than a list, the visual sitemap generator groups the same URLs into a navigable tree.

Frequently Asked Questions