Website Page Counter

What the counter counts

"How many pages does this site have?" sounds like one question, and the counter answers the most useful version of it: the number of URLs the site declares to search engines. Here is exactly what happens on a run:

1. 01

   Find the sitemap

   From a bare domain, robots.txt is checked first, then nine common sitemap paths. A pasted sitemap URL skips the discovery.

2. 02

   Walk the index

   Sitemap indexes are followed through their child files, up to 50 of them, including gzipped ones.

3. 03

   Count and de-duplicate

   Every declared URL counts once. Image and video sitemap entries that repeat a page URL per asset are de-duplicated.

4. 04

   Break the total down

   By first path segment, so the count arrives with a shape: how many pages live under /blog/, /products/, and so on. On very large sites the breakdown is computed from the first 5,000 URLs and labelled as sampled.

5. 05

   Fall back to a crawl when there is no sitemap

   Up to 100 pages by following internal links, clearly labelled, because a crawl count is a floor rather than a total.

The three page counts, and why they differ

A site does not have one page count. It has three, and the gaps between them are where the findings live:

On a healthy site the three run close together. The gaps are diagnoses:

• Declared far above indexed: Google is declining pages you are offering. The Pages report's exclusion reasons say why, and thin or duplicative content is the usual answer.
• Declared far above linked: the sitemap lists pages your own site never links to, which are orphan pages. The sitemap comparison tool finds them by diffing the two lists.
• Linked far above declared: the site generates URLs the sitemap ignores, usually parameters, pagination, or forgotten sections quietly spending crawl budget.

What your page count is telling you

The total is a headline; the section breakdown is the story. Three reads worth doing on your own result:

• Where the weight sits. If one section holds most of your pages, that is where your crawl attention goes, whether or not it is where your value sits. A 4,000-page tag archive dwarfing a 60-page product catalogue is a decision somebody should have made on purpose.
• Growth nobody chose. Sections grow by generation: tags, dates, filters, feeds. Any section whose count surprises you is worth opening, because surprise pages are rarely good pages.
• The earning share. Put the count next to how many pages actually earn clicks in Search Console. On most mature sites a small fraction of pages does almost all the earning, and knowing your ratio is the starting evidence for a content pruning pass.

Why the count matters for SEO

Page count is not a score, and more is not better. It matters as a denominator:

• Crawl attention is finite. Every declared page asks Google to spend crawl time on it. Thousands of pages that earn nothing are thousands of requests not spent on the pages that do.
• Quality is judged in aggregate. A site where most crawled pages are thin or generated reads as that kind of site. The count tells you how much surface you are asking Google to judge.
• Maintenance scales with pages. Every page is a future stale fact, broken link, and decayed ranking. The honest question is not "how many pages can we have?" but "how many can we keep good?"

From counting to knowing

The counter tells you how many pages exist. The follow-up questions, how many are indexed, how many earn traffic, and which ones deserve to stay, need your Search Console and analytics data joined to a crawl, which is exactly the join CrawlRaven runs: every page's search trend, engagement and technical findings in one ranked plan. The count is the denominator; the join fills in the numerator.

Frequently asked questions