Sitemap Crawler
Your sitemap parses cleanly. That does not mean the URLs inside it do. This crawler fetches each one and reports what Google will actually find.
Short answer
A sitemap crawler requests every URL your sitemap declares and reports what each one actually returns. A sitemap is a set of claims about which pages are worth indexing, and the claims go stale: URLs 404, redirect, carry a noindex, or name a different canonical. Google reads a sitemap full of contradictions as an unmaintained file and crawls it less often. This tool checks up to 100 URLs per run and groups them by the problem found.
The problem this tool exists to catch
Sitemaps are almost always generated, and generators rarely know what happened to a page after it was published. So the file stays syntactically perfect while its contents quietly rot.
- Deleted pages. A post is removed, the sitemap keeps listing it, and Google keeps fetching a 404.
- Migrated URLs. Slugs change, redirects go in, and the sitemap still points at the old address.
- Noindexed pages. Someone sets a page to noindex in the CMS, and the sitemap plugin never notices.
- Canonicalised duplicates. Paginated or parameter URLs get listed even though they canonicalise elsewhere.
- Slow pages. URLs that time out under a crawler get crawled less often than ones that respond quickly.
Why Google cares
Google treats sitemap accuracy as a signal about the site as a whole. A file full of dead and contradictory URLs gets fetched less frequently, which means every genuinely new page in it waits longer to be discovered. The cost is not a penalty, it is delay.
How to read the results
Each URL comes back with one status. Two of them are errors you should fix before anything else, and three are warnings worth understanding first.
- broken. The URL returns 4xx or 5xx. Remove it from the sitemap, or restore the page.
- noindex. The page loads but forbids indexing. Remove it from the sitemap, or drop the noindex.
- redirect. The URL bounces elsewhere. Replace the entry with its destination.
- canonical. The page names a different canonical URL. List the canonical target instead.
- timeout. No response within eight seconds. Usually a slow page rather than a broken one, but worth investigating.
- ok. Returns 200, is indexable, and is self-canonical. This is what every entry should be.
Fixing them in the right order
- 01
Remove the broken and noindexed URLs
These are contradictions, and they do the most damage to how much Google trusts the file.
- 02
Replace redirect entries with their destinations
Update at the source. Editing the generated file by hand only lasts until the next build.
- 03
Reconcile the canonical mismatches
Decide which URL is the real one, then make the sitemap and the canonical tag agree.
- 04
Fix the generator, not the file
Almost every problem here comes from a plugin or build step that does not check page state. That is where the durable fix lives.
- 05
Resubmit and recheck
Resubmit in Search Console, then run this crawler again in a few weeks to confirm the fix held.
What this free tool cannot tell you
It checks the first 100 URLs from one sitemap in a single request, which is enough to find systemic rot but not enough to audit a large site.
- It samples rather than covering everything, so a clean result on 100 URLs is evidence, not proof.
- It does not know which URLs Google has actually indexed, because that data lives in Search Console.
- It does not know which URLs earn traffic, because that lives in GA4.
- It cannot find pages missing from the sitemap, only problems with the ones that are in it.
- It does not execute JavaScript, so a noindex injected client side will not be detected.
For the file itself rather than its contents, run the sitemap validator. To pull the full URL list out for use elsewhere, use the sitemap extractor.
What each finding means, and how urgent it is
A sitemap is a set of claims about which pages are worth indexing. Every row below is a claim that turned out not to hold, ordered by how much it costs you.
| Finding | Why it matters | Fix |
|---|---|---|
| 404 or 410 | The file is telling Google to fetch pages that are gone | Remove the entry, or restore the page |
| Redirect | A sitemap should list final destinations only | Replace the entry with the destination URL |
| noindex | You are asking Google to index a page you told it not to | Remove the entry, or remove the noindex |
| Canonical points elsewhere | The listed URL is not the one that will be indexed | List the canonical URL instead |
| Different host or protocol | Invalid under the protocol | Move the URL into a sitemap on that host |
| 5xx | The page may be fine; the server was not | Re-check before acting on it |
Why Google cares
A sitemap full of contradictions is read as an unmaintained file, and unmaintained files get crawled less often. The cost is not a penalty, it is Google spending less of its attention on the one document you control completely.
Frequently asked questions
What does the sitemap crawler check?
It expands the sitemap, then fetches each URL and records its status code, whether it redirects and where to, whether the page carries a noindex directive in a meta tag or an X-Robots-Tag header, and whether its canonical tag points somewhere other than the sitemap entry itself.
How is this different from a sitemap validator?
A validator checks the file. This checks the URLs inside it. A sitemap can be flawless XML and still list a hundred dead pages, and Google will judge you on the second thing, not the first.
How many URLs does it crawl?
Up to 100 per run, taken from the start of the sitemap, within a 40 second budget. When a sitemap holds more, the result says so explicitly, so a clean score on a sample is never mistaken for a clean score on the whole file.
Why do redirects in a sitemap matter?
A sitemap is meant to be a list of canonical destination URLs. When it lists URLs that redirect, you are asking Google to crawl two pages to reach one, and repeating that across a large sitemap wastes real crawl budget.
Why is listing a noindex page a problem?
It is a direct contradiction. The sitemap says please index this, and the page says do not. Google resolves it in favour of the noindex, but the conflicting signals reduce how much it trusts the rest of the file.
What does the canonical warning mean?
The page loads fine but names a different URL as its canonical version. The sitemap entry is therefore not the URL that will be indexed, so the sitemap should list the canonical target instead.
Does a low health score hurt rankings?
Not directly. The cost is slower discovery and slower recrawling. Google reduces how often it fetches a sitemap it has learned to distrust, so new pages take longer to get found.
How often should I run this?
After any migration or bulk content change, and then quarterly. Sitemap rot is gradual: pages get deleted or set to noindex, and the generated sitemap keeps listing them until someone checks.
How many URLs does this tool check?
Up to 100 per run, chosen from the start of the sitemap. That is enough to tell whether a sitemap has a systemic problem, which is what you usually want to know. A file that is 40 percent redirects in its first hundred entries is not going to be clean in the remaining nine thousand.
Should I remove a URL from my sitemap as soon as it 404s?
Yes, from the sitemap. Whether to also restore or redirect the page is a separate decision that depends on whether it earned anything. What is never right is leaving the entry in place, because the sitemap is where you tell Google what you stand behind.
My CMS generates the sitemap. Can I still fix these?
Usually, yes, but at the source rather than in the file. Most generators include everything published regardless of its robots directive or canonical, so the fix is to correct the underlying page state and let the generator re-run. Editing a generated file by hand lasts until the next build.
Other sitemap tools
XML Sitemap Generator
Crawl any website and generate a valid XML sitemap you can download and submit. No signup, no software to install.
Use toolXML Sitemap Validator
Test any XML sitemap against the sitemaps.org protocol. Catches namespace errors, bad lastmod dates, size limits, and cross-domain URLs.
Use toolSitemap Checker
Check a sitemap end to end: the file against the sitemaps.org protocol, then a live check of the URLs inside it. One pass, one health score.
Use toolSitemap URL Extractor
Pull every URL out of a sitemap or sitemap index and download the list as CSV, TXT, or JSON. Handles nested indexes and gzipped files.
Use toolSitemap Index Generator
Paste your sitemap URLs and get a valid sitemap index file back, with optional lastmod dates. For sites past the 50,000 URL limit that split their sitemaps.
Use toolSitemap Comparison
Compare two sitemaps and see exactly which URLs were added, removed, and kept. Catch the pages a migration or a regenerated sitemap silently dropped.
Use toolSitemap Finder
Find where any website hides its sitemap. Checks robots.txt directives plus nine common paths, including WordPress and gzipped variants.
Use toolVisual Sitemap Generator
Turn any site into a visual tree of its structure. See how deep your pages sit and which sections carry the most URLs.
Use toolRead up on sitemaps
A tool tells you what is wrong. These explain what to do about it.
Screaming Frog Review 2026: Pricing, Features & Honest Verdict
Read guideHow to Perform a Technical SEO Audit in 2026 (Step-by-Step Guide)
Read guideTechnical SEO Audit Checklist 2026 (Free Template, 50+ Checks)
Read guideTerms this tool checks
- XML Sitemap
An XML sitemap is a file (typically at /sitemap.xml) that lists all the important URLs on your website along with metadata like last modification date, change frequency, and priority.
- Redirect Chain
A redirect chain occurs when a URL redirects to another URL, which then redirects to yet another URL, creating a series of sequential hops (e.g., Page A → 301 → Page B → 301 → Page C).
- Noindex
Noindex is a robots meta tag directive (meta name="robots" content="noindex") that tells search engines not to include a specific page in their search index.
- Canonical Tag
A canonical tag (rel="canonical") is an HTML element that tells search engines which version of a page is the preferred, authoritative copy when multiple URLs serve similar or identical content.
Check every URL, not the first hundred
CrawlRaven crawls your whole sitemap, then joins it to Search Console and GA4 so you can see which declared URLs Google has never indexed and which indexed pages your sitemap left out.