Sitemap Crawler
Your sitemap parses cleanly. That does not mean the URLs inside it do. This crawler fetches each one and reports what Google will actually find.
The problem this tool exists to catch
Sitemaps are almost always generated, and generators rarely know what happened to a page after it was published. So the file stays syntactically perfect while its contents quietly rot.
- Deleted pages. A post is removed, the sitemap keeps listing it, and Google keeps fetching a 404.
- Migrated URLs. Slugs change, redirects go in, and the sitemap still points at the old address.
- Noindexed pages. Someone sets a page to noindex in the CMS, and the sitemap plugin never notices.
- Canonicalised duplicates. Paginated or parameter URLs get listed even though they canonicalise elsewhere.
- Slow pages. URLs that time out under a crawler get crawled less often than ones that respond quickly.
Why Google cares
Google treats sitemap accuracy as a signal about the site as a whole. A file full of dead and contradictory URLs gets fetched less frequently, which means every genuinely new page in it waits longer to be discovered. The cost is not a penalty, it is delay.
How to read the results
Each URL comes back with one status. Two of them are errors you should fix before anything else, and three are warnings worth understanding first.
- broken. The URL returns 4xx or 5xx. Remove it from the sitemap, or restore the page.
- noindex. The page loads but forbids indexing. Remove it from the sitemap, or drop the noindex.
- redirect. The URL bounces elsewhere. Replace the entry with its destination.
- canonical. The page names a different canonical URL. List the canonical target instead.
- timeout. No response within eight seconds. Usually a slow page rather than a broken one, but worth investigating.
- ok. Returns 200, is indexable, and is self-canonical. This is what every entry should be.
Fixing them in the right order
- 01
Remove the broken and noindexed URLs
These are contradictions, and they do the most damage to how much Google trusts the file.
- 02
Replace redirect entries with their destinations
Update at the source. Editing the generated file by hand only lasts until the next build.
- 03
Reconcile the canonical mismatches
Decide which URL is the real one, then make the sitemap and the canonical tag agree.
- 04
Fix the generator, not the file
Almost every problem here comes from a plugin or build step that does not check page state. That is where the durable fix lives.
- 05
Resubmit and recheck
Resubmit in Search Console, then run this crawler again in a few weeks to confirm the fix held.
What this free tool cannot tell you
It checks the first 100 URLs from one sitemap in a single request, which is enough to find systemic rot but not enough to audit a large site.
- It samples rather than covering everything, so a clean result on 100 URLs is evidence, not proof.
- It does not know which URLs Google has actually indexed, because that data lives in Search Console.
- It does not know which URLs earn traffic, because that lives in GA4.
- It cannot find pages missing from the sitemap, only problems with the ones that are in it.
- It does not execute JavaScript, so a noindex injected client side will not be detected.
For the file itself rather than its contents, run the sitemap validator. To pull the full URL list out for use elsewhere, use the sitemap extractor.
Frequently Asked Questions
What does the sitemap crawler check?
It expands the sitemap, then fetches each URL and records its status code, whether it redirects and where to, whether the page carries a noindex directive in a meta tag or an X-Robots-Tag header, and whether its canonical tag points somewhere other than the sitemap entry itself.
How is this different from a sitemap validator?
A validator checks the file. This checks the URLs inside it. A sitemap can be flawless XML and still list a hundred dead pages, and Google will judge you on the second thing, not the first.
How many URLs does it crawl?
Up to 100 per run, taken from the start of the sitemap, within a 40 second budget. When a sitemap holds more, the result says so explicitly, so a clean score on a sample is never mistaken for a clean score on the whole file.
Why do redirects in a sitemap matter?
A sitemap is meant to be a list of canonical destination URLs. When it lists URLs that redirect, you are asking Google to crawl two pages to reach one, and repeating that across a large sitemap wastes real crawl budget.
Why is listing a noindex page a problem?
It is a direct contradiction. The sitemap says please index this, and the page says do not. Google resolves it in favour of the noindex, but the conflicting signals reduce how much it trusts the rest of the file.
What does the canonical warning mean?
The page loads fine but names a different URL as its canonical version. The sitemap entry is therefore not the URL that will be indexed, so the sitemap should list the canonical target instead.
Does a low health score hurt rankings?
Not directly. The cost is slower discovery and slower recrawling. Google reduces how often it fetches a sitemap it has learned to distrust, so new pages take longer to get found.
How often should I run this?
After any migration or bulk content change, and then quarterly. Sitemap rot is gradual: pages get deleted or set to noindex, and the generated sitemap keeps listing them until someone checks.
Other sitemap tools
XML Sitemap Generator
Crawl any website and generate a valid XML sitemap you can download and submit. No signup, no software to install.
Use toolXML Sitemap Validator
Test any XML sitemap against the sitemaps.org protocol. Catches namespace errors, bad lastmod dates, size limits, and cross-domain URLs.
Use toolSitemap URL Extractor
Pull every URL out of a sitemap or sitemap index and download the list as CSV, TXT, or JSON. Handles nested indexes and gzipped files.
Use toolSitemap Finder
Find where any website hides its sitemap. Checks robots.txt directives plus nine common paths, including WordPress and gzipped variants.
Use toolVisual Sitemap Generator
Turn any site into a visual tree of its structure. See how deep your pages sit and which sections carry the most URLs.
Use toolCheck every URL, not the first hundred
CrawlRaven crawls your whole sitemap, then joins it to Search Console and GA4 so you can see which declared URLs Google has never indexed and which indexed pages your sitemap left out.