The Crawl Stats Report: What Googlebot Actually Did on Your Site for 90 Days
Crawl stats is buried in Settings and answers questions no other report can. The four breakdowns, the patterns worth acting on, and who really has a crawl budget problem.
Crawl stats sits in Settings rather than the sidebar, which is why most people have never opened it. It holds 90 days of what Googlebot actually did, and no other report answers those questions. What matters in it:
- Where it is: Settings, then Crawl stats. Available on domain properties and root-level URL-prefix properties, not on a property scoped to a subfolder.
- The one causal relationship: Average response time up, crawl requests down. Google reduces its crawl rate when your server struggles, so this is a hosting problem that presents as an SEO one.
- Four breakdowns: By response, by file type, by purpose, and by Googlebot type. The purpose split, Discovery against Refresh, is the one people never look at and often should.
- Host status is the fastest check: robots.txt fetch, DNS resolution, server connectivity. A robots.txt returning 5xx makes Google pause crawling the whole site until it can read it.
- Crawl budget is not your problem: Google says sites under a few thousand URLs are crawled efficiently. Below that scale, read this report for errors rather than for volume.
There is no target number for crawl requests. Every finding here is a pattern over time or an error, never the headline figure.
Crawl stats is a free Google report and this is a read of it. CrawlRaven's crawl covers the half it cannot show you, which is what your site is actually serving, and that comparison is the last section. Try CrawlRaven free: 1 site, no credit card →
Crawl stats answers a question no other Search Console report touches: not what Google decided about your pages, but what Googlebot physically did on your server for the last 90 days.
It is also filed under Settings rather than in the sidebar with the reports, which is most of the reason people who have used Search Console for years have never opened it.
What crawl stats holds, and where to find it
Open the property, go to Settings, then Crawl stats. Two things about availability catch people out before they see a single number.
- Ninety days, not sixteen months. This is the shortest window in Search Console, and unlike Performance data there is no bulk export to preserve it. Your screenshots are the archive.
- Property scope matters. It is available on domain properties and on URL-prefix properties at the root of a host, and not on a property scoped to a subfolder. If the option is missing, check which kind of property you are on before concluding anything.
Which property type you created is a decision that keeps having consequences, and the tradeoffs are laid out in the Search Console guide.
The three headline numbers, and the one that matters
The report opens with total crawl requests, total download size and average response time. Two of those are context and one is a finding.
- Total crawl requests has no good value. It depends on your site's size, how often it changes and how fast it responds, so it is not comparable to anyone else's and has no target.
- Total download size is mostly interesting when it moves without the request count moving, which usually means your pages got heavier.
- Average response time is the one to watch. It is the input Google uses to decide how hard it can crawl you, which makes it the only number here with a direct consequence.
Read the three together and one pattern does almost all the diagnostic work in this report, covered below.
Host status: the 20-second check
Above the charts, host status reports whether Google could do three things over the last 90 days: fetch robots.txt, resolve your DNS, and connect to your server.
All three have to work for crawling to happen at all, which makes this the fastest useful check in Search Console. A failure here explains problems that look inexplicable everywhere else.
The robots.txt line deserves particular attention, because its failure mode is counterintuitive:
- A 200 gives Google rules. It fetches the file and follows it.
- A 404 means no rules apply. Google crawls freely, which is fine.
- A 5xx is ambiguous, so Google stops. It cannot tell whether you meant to disallow something, so it pauses crawling the site rather than risk it.
That last case is worth designing against explicitly. Check what yours actually returns with the free robots.txt tester, and if you are writing one from scratch the robots.txt generator produces a file that allows the crawlers you want.
The four breakdowns
Every request is classified four ways at once. Each cut answers a different question, and one of them is routinely ignored.
The crawl stats breakdowns
The purpose split is the underused one. Discovery means Google fetched a URL it had never seen; Refresh means it re-checked one it knew about.
A site publishing weekly with discovery near zero is not being crawl limited. It is failing to tell Google that new pages exist, which is an internal linking or sitemap problem and a much cheaper fix than a crawl budget project.
Patterns worth acting on
Nothing in this report is actionable as a single number. Everything useful is a relationship between two lines or an error rate that should be zero.
What each crawl stats pattern means
The first row is the causal chain worth internalising. Google reduces its crawl rate when your server responds slowly, so a falling request count beside a rising response time is not two problems. It is one problem and its consequence, and the fix is infrastructure rather than SEO.
The redirect row is the one that shows up after a migration. Every hop is a request that fetched no content, and the fix is updating internal links to point at the final URL rather than leaving the redirect to absorb it.
The free redirect checker traces a chain end to end, and tracking a site migration covers the wider measurement.
Who actually has a crawl budget problem
Crawl budget is the least useful popular concept in technical SEO, because the people who read about it overwhelmingly do not have the problem it describes.
Google defines it as two things multiplied: how much it wants to crawl your site, and how much your server can take. Both have to bind before it is a constraint.
Who actually has a crawl budget problem
The bottom row is the one people miss. Crawl capacity is set by how well your server responds, so a slow host caps the crawl on a 500-page site just as firmly as on a large one. Size is not the only entry ticket.
For the genuinely large-site case, the definition and the levers are in the crawl budget entry, and the tooling that goes deeper than this report is compared in the technical audit tools roundup.
I have watched crawl budget consume weeks on sites with four hundred pages. It is an attractive problem: technical enough to feel serious, measurable enough to make a chart, and vague enough that you can always find something to optimise.
The honest test is the response time line. If your server is fast and your error rate is zero, Google is crawling everything it wants to and the constraint is somewhere else entirely, usually in whether the pages deserve to be indexed at all.
What crawl stats will not tell you
The report is aggregate by design, and every limit below is the same limit: it counts requests without naming URLs.
- No URL lists. You get a redirect share and an error rate with nothing to click through to. Which pages caused them is not in the report.
- Only 90 days. The shortest window in the product, with no export and no BigQuery equivalent. Take a screenshot before a migration.
- Google's crawlers only. GPTBot, ClaudeBot and PerplexityBot appear in your server logs and nowhere here, which is part of why there is no Search Console for ChatGPT.
- No API. Crawl stats is not exposed through the Search Console API, so it cannot be monitored programmatically.
Server logs cover most of those gaps and are the natural next step once this report tells you something is wrong but not where.
A quarterly read, and when to look sooner
Crawl stats rewards a fixed, infrequent routine. Four checks, ten minutes, once a quarter.
- Host status first. Three green ticks and you can read the rest calmly. Anything else outranks everything below it.
- Response time against requests. One glance at the two lines together. Diverging is the finding.
- By response, looking for 5xx. Any sustained share is urgent regardless of how small the percentage looks.
- Discovery against refresh, if you publish regularly. Low discovery on an active site is a linking problem worth an hour.
Look sooner than quarterly in three situations: after a migration, after a hosting change, and when indexing has slowed for no reason you can find in the page indexing report. In that third case, crawl stats is often where the answer was all along.
Key Takeaways
- →Where it lives: Settings, then Crawl stats. Domain properties and root-level URL-prefix properties only, which is why it looks missing on a subfolder property.
- →Ninety days, no export: The shortest window in Search Console, with no bulk export and no API. Screenshot it before a migration, because that is the only archive you will have.
- →The one causal chain: Average response time up, crawl requests down. Google throttles its crawl when your server struggles, so this is a hosting problem presenting as an SEO one.
- →Check host status first: robots.txt, DNS and server connectivity. A robots.txt returning 5xx pauses crawling of the whole site, because Google cannot confirm what it may fetch.
- →Read the purpose split: Discovery near zero while you publish weekly is not a crawl limit. It is Google failing to learn your new pages exist, which is linking or sitemaps.
- →Crawl budget is a large-site problem: Google says sites under a few thousand URLs are crawled efficiently. The exception is any site with a slow server, where capacity binds regardless of size.
- →It counts requests, it never names URLs: You get a redirect share and an error rate with nothing to click. Finding the pages behind them needs a crawl of your own site or your server logs.
Primary sources
Frequently asked questions
Where is the crawl stats report in Google Search Console?
Under Settings rather than in the main sidebar: open the property, go to Settings, then Crawl stats. It is available for domain properties and for URL-prefix properties at the root of a host, and not for a property scoped to a subfolder, which is one reason people conclude it does not exist.
What does the crawl stats report show?
Ninety days of Googlebot activity on your site: total crawl requests, total download size and average response time, plus host status for robots.txt, DNS and server connectivity. Requests break down four ways, by response code, by file type, by purpose (discovery or refresh) and by Googlebot type.
What is a good number of crawl requests?
There isn't one. The absolute figure depends on your site's size, how often it changes and how fast it responds, so it is not comparable to anyone else's and there is no target to hit. Every real finding in this report is a change over time or an error rate, never the headline number on its own.
Why did my crawl rate drop in Search Console?
Check average response time on the same chart first. Google lowers its crawl rate when a server responds slowly or returns errors, so a falling request count next to a rising response time is cause and effect rather than two coincidences. If response time is flat, a drop usually just means less has changed on the site.
What is crawl budget, and do I need to worry about it?
Crawl budget is the combination of how much Google wants to crawl your site and how much your server can take. Google says sites under a few thousand URLs are crawled efficiently and do not need to think about it. It becomes a genuine constraint on very large sites, on sites with faceted navigation generating URLs endlessly, and on any site with a slow server.
What does host status mean in crawl stats?
It reports whether Google could fetch robots.txt, resolve your DNS and connect to your server over the last 90 days. All three have to work for crawling to happen at all, which makes this the fastest useful check in the report: a failure here explains problems that look inexplicable in every other report.
Why does robots.txt returning a 5xx stop my whole site being crawled?
Because Google cannot confirm what it is allowed to fetch. A 200 gives it rules and a 404 means no rules apply, but a server error is ambiguous, so Google pauses crawling rather than risk fetching something you meant to disallow. Make robots.txt return 200 or 404, never a 5xx.
What is the difference between discovery and refresh crawling?
Discovery is Google fetching a URL it has not seen before. Refresh is re-checking one it already knows. The split tells you whether Google is finding your new content: discovery near zero while you publish weekly usually points at internal linking or a stale sitemap rather than at a crawl limit.
Can I increase my crawl rate in Search Console?
Not directly any more. The old crawl rate setting only ever let you slow Google down, and the practical levers are indirect: make the server faster and more reliable, stop wasting requests on redirects and error pages, and give Google fewer low-value URLs to spend the crawl on.
Does crawl stats show AI crawlers like GPTBot?
No. This report covers Google's own crawlers only, split by Googlebot type. Anything from OpenAI, Anthropic or Perplexity appears in your server logs and nowhere in Search Console, which is one of the reasons logs remain worth keeping.
15+ years of growing SaaS websites through SEO | Author, 200-Point Audit Checklist
Aditi has spent 15+ years helping SaaS companies scale organic traffic through technical SEO and content strategy. She is the author of the CrawlRaven 200-Point Audit checklist used by agencies and in-house teams to systematically improve search performance.