Website URL Extractor
Enter a domain and get every URL on the site. The crawler follows internal links the way a search engine does, lists what it finds with the crawl depth of each page, and exports the lot as CSV, TXT or JSON.
Short answer
A website URL extractor crawls a live site by following its internal links and returns every URL it discovers, whether or not the site has a sitemap. This one starts at the homepage, follows links breadth-first across up to 100 pages, respects robots.txt, records the click depth of every page, and exports the list as CSV, TXT or JSON. It reports what is actually linked on the site, which is a different list from what the sitemap declares, and the difference between the two is where orphan pages and undeclared pages hide.
Two ways to get every URL on a site
There are two honest ways to build a list of a site's URLs, and they answer different questions. You can read the sitemap, which is the list of pages the site says it has. Or you can crawl the site by following links, which is the list of pages the site actually connects to. This tool does the second. The sitemap URL extractor does the first.
Reading the sitemap wins on speed and completeness: one file, no crawling, and on a well-maintained site it lists everything, including pages nothing links to. Crawling wins on truth: it works on sites with no sitemap at all, it sees the URL variants and generated pages a sitemap politely omits, and it tells you how each page is actually reached, which a sitemap cannot.
The interesting result is not either list. It is the disagreement between them. A URL in the crawl but not the sitemap is a page you link to and never declared. A URL in the sitemap but not the crawl is a page you declared and no longer link to. Both are silent problems, and neither shows up while you only look at one list. If you do not know where the sitemap lives, the sitemap finder will locate it first.
What a crawl finds that a sitemap hides
A sitemap is a claim, written by whoever configured the CMS plugin, and it drifts from reality in both directions. The crawl catches both kinds of drift.
In one direction are the pages your site links to that the sitemap never mentions: tag archives, paginated lists, campaign landing pages someone published outside the CMS, and old URLs still linked from a footer. Search engines find these through your links whether you declare them or not, so they are part of your crawled site either way. Better to know the list than to let a crawler decide what it means.
In the other direction are the pages the sitemap declares that no link reaches. These are orphans, or close to it: they depend entirely on the sitemap for discovery, they collect no internal authority, and they are typically the pages whose rankings decay first. A sitemap reader cannot see this problem at all, because from inside the sitemap every page looks equally well connected.
| Source | Includes | Misses | Best for |
|---|---|---|---|
| Crawl (this tool) | Every page reachable by following links, with the click depth of each | Orphan pages nothing links to; links rendered only by client-side JavaScript | Seeing the site as a search engine first meets it, sitemap or not |
| Sitemap extractor | Every URL the site declares, with lastmod where present | Undeclared pages: tag archives, pagination, forgotten but still-linked URLs | Fast complete lists from sites that maintain their sitemap |
| Both, diffed | The disagreement: linked-but-undeclared and declared-but-unlinked pages | Pages absent from both lists, which are invisible to search engines too | Finding orphan pages and sitemap gaps in one pass |
Neither list is the site
A page missing from both the crawl and the sitemap is unreachable by links and undeclared, which means search engines have no route to it either. If such a page matters, the fix is a link to it, not just a sitemap entry.
How to extract every URL from a website
Five steps, and the last one is where the findings are. Extracting the list is the easy part; comparing it against what the site claims about itself is the audit.
- 01
Enter the domain
Paste the site's address, such as https://example.com. The crawl always starts at the homepage, because that is where a search engine starts too, so the depth numbers you get back mean the same thing they mean to Google.
- 02
Let the crawl finish
The crawler follows internal links breadth-first, fetching up to 100 pages within a 45 second budget and respecting robots.txt as it goes. If the site is larger than 100 pages, the result says so explicitly rather than presenting a partial list as a complete one.
- 03
Read the depth column
Depth is the number of clicks from the homepage to each URL. Depth 0 is the homepage, depth 1 is everything it links to directly. Sort by this column and look at the bottom: pages at depth 4 or more are the ones your own site is hiding.
- 04
Download the CSV
The CSV carries url, depth and lastmod columns, so it drops straight into a spreadsheet or a script. TXT gives you one URL per line for tools that want plain lists, and JSON keeps the full structure.
- 05
Diff the list against your sitemap
Run the same domain through the sitemap comparison tool. URLs in the crawl but not the sitemap are pages you link to but never declared. URLs in the sitemap but not the crawl are pages you declared but no longer link to. Both lists are findings.
The sitemap comparison tool does the diff in step five for you: give it the crawl list and the sitemap and it returns the URLs unique to each side.
What the depth column tells you
Every URL in the results carries a depth: the number of clicks from the homepage to that page. It looks like a minor detail and it is one of the most useful columns in the export, because depth is how your internal linking spends the site's crawl budget.
Search engines discover a site the same way this tool does, outward from the pages they already know. A page at depth 1 gets found on the first pass, revisited often, and inherits authority from the homepage directly. A page at depth 4 or more gets found late, crawled less, and reached only through a chain of intermediate pages. On a large site, deep pages can go weeks between crawls while shallow ones are fetched daily.
Reading the column takes about a minute:
- Important pages at depth 3 or more are under-linked. Add links from shallower pages that already have authority
- A cluster of URLs at the same deep level usually shares one cause, such as pagination or a buried archive
- A money page deeper than a tag archive means your linking priorities are inverted
- If the crawl hit the 100-page cap, the pages it never reached are deeper still, which is itself the finding
Depth problems are cheap to fix relative to what they cost. A link from the homepage or a hub page moves a page from depth 4 to depth 1 in one edit, and the crawl and authority benefits follow from there.
What this tool does
Live crawl
Follows real internal links on the live site, not a cached copy or a declared list
100 pages free
Crawls up to 100 pages per run, and says so when a site has more
Robots.txt respected
Reads the site's robots rules first and reports every URL skipped because of them
CSV, TXT and JSON export
One click to a spreadsheet-ready CSV, a plain URL list, or structured JSON
Depth per URL
The click distance from the homepage to every page, the column most audits start from
Duplicate-free list
Each URL appears once, with fragments stripped and non-HTML assets filtered out
No signup
Paste a domain and crawl. No account, no email, no export paywall
Works without a sitemap
Link-following discovery needs no sitemap, so it works on sites that never made one
Frequently asked questions
How do I get a list of every URL on a website?
Enter the domain above and the tool crawls the live site by following internal links, starting from the homepage, and returns every URL it finds as a list you can copy or download as CSV, TXT or JSON. The free crawl covers up to 100 pages. For a complete list on a larger site, combine it with the sitemap extractor and diff the two.
How is this different from the sitemap URL extractor?
This tool discovers URLs by crawling the live site and following links, so it reports what is actually linked. The sitemap URL extractor reads the sitemap file, so it reports what the site declares. The two lists routinely disagree, and the disagreement is the useful part: pages linked but missing from the sitemap, and pages declared but no longer linked from anywhere.
Why does it find fewer URLs than my sitemap?
Because some pages in your sitemap are not reachable by following links from the homepage. Those are orphan or near-orphan pages: declared to search engines but cut off from your internal linking, which means they get little crawl attention and no internal authority. Finding them is one of the main reasons to run a crawl at all.
Why does it find more URLs than my sitemap?
Because your site links to pages the sitemap never mentions. Common causes are tag and pagination pages generated by a CMS, old pages that were dropped from the sitemap but never unlinked, and URL variants such as trailing-slash duplicates. Each extra URL is either a page worth declaring or a link worth removing.
What is crawl depth?
Crawl depth is the minimum number of link clicks from the homepage to a page. The homepage is depth 0, pages it links to directly are depth 1, and so on. Depth is a practical measure of how findable a page is: search engines discover deep pages later, revisit them less often, and pass them less internal authority.
Does it respect robots.txt?
Yes. The crawler fetches the site's robots.txt first and skips any path it disallows, and the result reports how many URLs were skipped for that reason. If the whole site is disallowed, the crawl returns an error rather than ignoring the file.
Why did it stop at 100 pages?
The free crawl is capped at 100 pages and 45 seconds so it stays fast and polite to the site being crawled. When a site is bigger than that, the result says how many URLs were discovered in total, so you know the list is partial rather than assuming it is complete.
Does it work on JavaScript sites?
Partly. The crawler reads the HTML the server returns and follows the links found in it, which covers server-rendered and statically generated sites completely. Links that only exist after client-side JavaScript runs are not seen, so a fully client-rendered site may return only the URLs present in its initial HTML. That limitation applies to many crawlers, which is one reason server rendering matters for SEO.
Can I extract URLs from someone else's site?
Yes, for any publicly reachable site. Crawling public pages by following links is what every search engine does, and this tool does it politely: it obeys robots.txt, keeps concurrency modest and stops at 100 pages. It is useful for studying a competitor's structure or auditing a client site before you have access.
Why are some pages missing from the results?
Four common reasons. The page is more than 100 pages deep into the crawl order, so the cap cut it off. Nothing links to it, so a link-following crawler cannot reach it. Robots.txt disallows its path. Or the link to it only renders client-side in JavaScript. The result flags the first and third cases explicitly.
What do I do with the CSV?
The CSV has url, depth and lastmod columns, which makes it the starting point for most audits. Sort by depth to find buried pages, diff the url column against your sitemap to find orphans and undeclared pages, paste the list into a rank tracker or a redirect map, or keep dated exports to see what a site added and removed over time.
Does it follow subdomains?
No. The crawl stays on the exact host you enter, so a crawl of example.com will list links to blog.example.com without crawling them. If the subdomain is a site in its own right, run it as a separate crawl and you will get its own list with its own depth numbers.
Is it legal and polite to crawl a website?
Crawling publicly accessible pages is how the web is indexed, and this tool follows the established conventions: it identifies itself with a user agent, obeys robots.txt, limits itself to 100 pages and keeps concurrency low. That is a lighter touch than a normal search engine visit. It never bypasses logins or fetches pages a site has asked crawlers to skip.
Other crawl and indexing tools
Redirect Chain Checker
Trace every redirect hop with status codes and response times. Detect loops, long chains, protocol downgrades, and 302 misuse.
Use toolPage Indexability Checker
Find out why a page is not indexed. Checks status codes, noindex tags and X-Robots-Tag headers, robots.txt rules, canonicals, sitemap membership, and rendered content.
Use toolBroken Link Checker
Find broken internal and external links on any page. Checks up to 50 links for 404 errors, redirects, and timeouts.
Use toolRobots.txt Tester
Analyze robots.txt to see which AI crawlers and search engines are blocked, find sitemaps, and identify access issues.
Use toolRobots.txt Generator
Build a correct robots.txt from presets for WordPress, ecommerce, and AI crawlers. Decide which bots to allow, add your sitemap, copy the file.
Use toolThis lists the URLs. CrawlRaven tells you which ones matter
CrawlRaven crawls your whole site, then joins every page to its Search Console and GA4 data, so the same list comes back ranked by the clicks, impressions and revenue each page is worth, with the orphans and the buried pages already flagged.