429 Status Code: What Happens When Googlebot Gets One
A 429 means Too Many Requests. Serve one to Googlebot and crawling slows across your whole host within seconds, and the recovery is measured in weeks.
A 429 status code means Too Many Requests: the server is rate-limiting the client. When that client is Googlebot, the response is also an instruction to crawl your whole host less. What matters:
- Google reads 429 as overload: Google's documentation says its crawlers treat 429 as a signal that the server is overloaded, and that 5xx and 429 responses make them temporarily slow down.
- The limit is shared: As reported from Search Central Live in Barcelona, one crawl rate limit per host covers Search, Ads, Shopping and Images crawling.
- Three inputs move it: Connection time, time to first byte, and 429 or 5xx responses, according to John Campbell's recap for ROAST.
- Down in seconds, up in weeks: Reported timings: capacity can drop in seconds, typically takes 4 hours to 1-2 weeks to update, and 1-3 weeks in recovery.
- A CAPTCHA on a 200 is worse: A challenge page that returns 200 OK looks like a real page with no content, which Google treats as a soft 404.
Serve a 429 or 503 on purpose when the server really is struggling, keep it under 1 to 2 days, and make sure your WAF is not serving one by accident.
This post is about what your server tells Googlebot under load. CrawlRaven joins Search Console, GA4 and a 200-point crawl into one ranked plan, so slow responses and error pages show up next to the traffic they put at risk. That fit is the last section. Try CrawlRaven free: 1 site, no credit card →
A 429 status code means Too Many Requests: the client has sent too many requests in a given amount of time, and the server is rate-limiting it. That is the whole definition, and every result on page one for the query stops there.
None of them covers the client that matters most to your traffic. When Googlebot gets a 429, it does not only wait and retry. It crawls your whole host less, and getting that capacity back takes far longer than losing it did.
What does a 429 status code mean?
A 429 status code means the server is refusing a request because the client has made too many in a short period. MDN's reference calls the mechanism rate limiting, and the code was defined in RFC 6585. Three details matter for the rest of this post.
- It is a 4xx code. Formally it blames the client, not the server. Google treats it differently from every other 4xx, which is the next section.
- It can carry a Retry-After header. The server may say how long the client should wait before trying again.
- The limit is whatever the server decides. MDN notes it may be server-wide or per resource, and is typically based on the client's IP address.
The query gets about 2,400 US searches a month (DataForSEO, US, October 2026), mostly from developers who hit a rate limit on an API. If that is you, slow your requests and respect Retry-After. If you run the server, keep reading.
The server is fine, but it thinks you are asking too often and wants you to back off for a while. A 503 says something different: the server itself cannot handle requests right now.
What a 429 does when the client is Googlebot
Googlebot treats a 429 as a statement about your server's health. Google's HTTP status code documentation says all 4xx errors except 429 are treated the same, and that its crawlers read 429 as a signal that the server is overloaded.
The same page spells out what follows from that signal:
- Crawling slows. 5xx and 429 responses prompt Google's crawlers to temporarily slow down.
- The response body is thrown away. Any content returned with a 5xx status code is ignored.
- Indexed pages survive for a while. Already indexed URLs are preserved in the index, but eventually dropped if the errors persist.
- Recovery is gradual. Once the server answers with 2xx again, Google gradually increases the crawl rate for the site.
So a 429 is a brake, not a removal request. The problem is how wide the brake applies. Google's page on reducing crawl rate says these codes reduce the crawl rate across the whole hostname, not only for the URLs that returned them.
One crawl rate limit, shared by four crawlers
The hostname-wide effect got more specific at Search Central Live Deep Dive in Barcelona, which ran from 30 September to 2 October 2026. I was not there, and Google published no official recap, so what follows comes from John Campbell's Day 1 recap for ROAST.
According to that recap, Cherry Prommawin's session described one central crawling system for Search, Ads, Shopping and Images, and split crawl budget into two parts: crawl demand, and a crawl rate limit also called hostload.
One allowance, four crawlers
Two reported details about the crawl rate limit change how I read a 429:
- It is shared. The limit covers all of Google's crawlers for your host, so Ads and Images crawling comes out of the same allowance as Search.
- Three inputs drive it. Connection time, time to first byte, and 429 or 5xx status codes. If any of them rise, Google slows the crawl.
Read together, those mean a rate limit aimed at one noisy crawler is not contained. If the recap is accurate, a burst of 429s shrinks the allowance that Shopping and Ads fetches draw on too.
One line attributed to the same session in Campbell's notes, without surrounding context, was: “You can buy crawl budget, get a better and faster server.” Google's own crawl budget guide makes the same point more quietly, listing “Add more server resources” as a way to get more crawling.
Status codes and what Googlebot does with each
The reference version, built from Google's documentation. The middle column is what Google says its crawlers do. The right-hand column is my recommendation.
| Status code or condition | What Googlebot does | What you should do |
|---|---|---|
| 200 with a CAPTCHA or challenge page | Processes the response as content. If it looks empty or like an error, Search Console reports a soft 404. | Never challenge Google's crawlers on a 200. Let them through, or send an honest 429 or 503. |
| 429 Too Many Requests | Treats it as a sign the server is overloaded and temporarily slows crawling. | Fine for a real spike, for hours. Find and fix any WAF rule sending it by accident. |
| 500 Internal Server Error | Decreases the crawl rate for the site and ignores the content. URLs that persistently error are removed from the index. | Treat as a bug. Fix the application error before it persists. |
| 503 Service Unavailable | Handled as a 5xx server error: crawling slows, and the crawl rate rises again gradually once 2xx responses return. | The right code for maintenance and overload. Keep it under 1 to 2 days. |
| Slow time to first byte, no error | Crawl capacity limit goes down when the site slows down, so Google crawls less. | Fix server response time. This is a hosting and caching job, not a robots.txt one. |
| DNS error, timeout or connection reset | Treated similarly to 5xx. Crawling starts slowing immediately, and unreachable indexed URLs are removed within days. | Check firewall rules for blocked Google IP addresses, then your DNS records. |
The DNS row comes from Google's network and DNS errors page, which says timeouts, connection resets and DNS errors are treated similarly to 5xx server errors. The slow-response row comes from the crawl budget guide. Slowness is the quiet one: no error is logged anywhere, and page speed work rarely looks at the server's first byte.
The worst answer: a 200 with a CAPTCHA
The first row of that table is the one I would fix before anything else. Bot protection often answers a suspicious request with a challenge page, and many setups send that page with a 200 OK.
As reported from the crawl errors session in Barcelona, Google called this out as a growing source of soft 404s: a CAPTCHA page can return 200 OK, so Google sees a page that looks fine but has no real content. Here is why that is worse than a 429:
- A 429 is honest. Google slows down, keeps what it has indexed for now, and comes back.
- A 200 is a claim of success. Google's documentation says a 2xx response is considered for processing, and that content suggesting an error or an empty page is reported as a soft 404.
- Soft 404s keep costing you. The crawl budget guide says soft 404 pages continue to be crawled and waste your budget.
With a 429, Google knows it did not see your page. With a challenge on a 200, Google thinks it did, and what it saw was a puzzle. That shows up later as pages stuck in the states covered in crawled vs discovered, currently not indexed.
When your WAF or CDN sends the 429 for you
Most 429s that reach Googlebot are not a decision anyone made. The Barcelona recap reports Google saying CDNs are blocking more bot traffic, with a new Cloudflare setup that might end up blocking Googlebot as the example. Those blocks show up as HTTP errors.
The “What would you do?” session on Day 1 put a version of this to John Mueller and Gary Illyes. As Campbell describes the scenario:
- The setup. During a flash sale, the WAF mistakes crawl traffic for an attack and serves 429 and 503 errors to Googlebot.
- The damage. Crawl rate drops 90% across the domain.
- Illyes' reported answer. Check server usage and fix the WAF.
- Mueller's reported answer. Do nothing.
Reading the recap, I do not think those two answers disagree as much as they look. Checking server usage tells you whether the load was real. If it was, the 429 did its job and Google's crawl rate comes back on its own, which is the “do nothing” answer.
If the server was idle and the WAF panicked, doing nothing leaves the same rule armed for the next sale. That is the case worth an hour of someone's time.
A related trap is worth knowing about. Blocks do not only happen on your own hostname, and a CDN or subdomain that serves your scripts has its own rules, as covered in robots.txt on third-party hosts.
When a 429 or 503 is the right thing to serve
Serving an error to Googlebot on purpose is sometimes correct. Google documents it: to urgently reduce crawling for a couple of hours or 1 to 2 days, return 500, 503 or 429 instead of 200 to the crawl requests.
What to serve Googlebot, by situation
The matrix above compresses five situations. In prose:
- Genuine overload. Serve 503 or 429. Google says you can add a Retry-After header to either, and it raises the crawl rate automatically once the errors fall.
- Planned maintenance. Serve 503. Content from a 5xx response is ignored, so the maintenance page is never read as your content.
- A WAF that cannot tell Googlebot from an attack. Fix the rule and serve Googlebot a 200. A 429 here cuts crawling across the hostname for a problem the server never had.
- A bot challenge. Never on a 200, for the soft 404 reasons above.
The fifth situation is pausing crawling site-wide, and it came up in Barcelona. In the stale-cache scenario, metadata had changed on 500,000 product pages, the CDN edge cache was set to 30 days, and the purge API was rate-limited. Illyes' reported answer was to do nothing. Mueller's was to serve a 503 on robots.txt to pause crawling.
Google's robots.txt documentation explains why that works, and its limit. For the first 12 hours of a 5xx on robots.txt, Google stops crawling the site. After that it falls back to the last good version of the file, so this is a short pause, not a switch.
How to check whether Googlebot is being rate-limited
You cannot see this from a browser, because your browser is not the client being limited. Work through four places, in this order.
- Crawl stats, host status. In Search Console, open Settings, then Crawl stats. Host status covers robots.txt fetching, DNS resolution and server connectivity over 90 days.
- Crawl stats, by response. Look at the share of requests that did not return OK, and at average response time against total requests. The crawl stats report guide walks through each breakdown.
- Server and CDN logs. Filter to Google's crawlers and count responses by status code per hour. Log file analysis is the only place you see the 429s themselves, with URLs attached.
- WAF and bot rules. Read every rate-limit, challenge and bot-management rule. Google's own debugging advice is to make sure Google IP addresses are not blocked by an overly broad firewall rule.
Compare origin logs with CDN logs if you have both. A 429 that appears at the edge and never reaches the origin was sent by a rule, not by a struggling server. For a quick look at what a single URL returns, the free HTTP header checker shows the status code and response headers, including any Retry-After.
How long recovery takes
Recovery is slower than the drop, by a wide margin. On Day 3 in Barcelona, Gary Illyes showed timing tables for Google's processes, reproduced by Search Engine Roundtable. The crawl capacity row is the relevant one here.
Seconds down, weeks back up
- Down: crawl capacity can drop in seconds when Google backs off, for example if your server struggles.
- Typical update: 4 hours to 1-2 weeks.
- Slowest: 1-3 weeks, in recovery.
- Google's documented window: do not serve these errors for longer than 1 to 2 days, because the same URL returning them for multiple days may be dropped from the index.
Treat the Barcelona figures as indicative. Illyes told Search Engine Roundtable the slides were “an exercise to see if the audience can relate to the numbers we pulled internally.” The asymmetry is the useful part: a WAF rule that misfires for an afternoon can cost weeks of normal crawling.
That delay stacks onto everything downstream, since a page cannot be indexed until it is crawled. The full set of reported timings is in how long Google takes to index a page.
What this means for you
The practical takeaways differ by who owns the server.
- In-house SEO. Ask whoever runs the WAF what Googlebot gets during a traffic spike. Get the answer before the next sale, not after it.
- Agency or consultant. Add host status and the by-response breakdown to every audit. A client who changed CDN recently is the first place to look.
- Developer. Return real status codes. A challenge or error page on a 200 hides the problem from every monitoring tool as well as from Google.
- Founder on a small site. Crawl budget is rarely your constraint, but a slow or erroring server is. Google's guide is written for large sites and for sites of 10,000 or more pages that change daily.
Key Takeaways
- →429 means Too Many Requests: The server is rate-limiting the client. For most clients that is the end of the story.
- →Googlebot reads it as overload: Google's documentation treats 429 differently from every other 4xx: crawlers slow down across the whole hostname.
- →The allowance is shared: As reported from Barcelona, one crawl rate limit per host covers Search, Ads, Shopping and Images.
- →Speed counts as much as errors: Connection time and time to first byte are reported inputs alongside 429 and 5xx responses.
- →Never challenge on a 200: A CAPTCHA page returning 200 OK becomes a soft 404. An honest 429 or 503 is the better failure.
- →Keep deliberate errors short: Google's window is a couple of hours to 1 to 2 days. Reported recovery runs up to 1-3 weeks.
Where CrawlRaven fits
Crawl stats tells you an error share went up. It does not tell you which URLs, or whether they are pages anyone visits. We built CrawlRaven to join Search Console, GA4 and a 200-point crawl into one ranked plan, so error responses and slow pages are listed with the clicks and sessions behind them.
It will not show you what your WAF served Googlebot last Tuesday. Only your logs can do that. What it gives you is the order to fix things in once you know something is wrong.
Sources used in this post
- MDN: 429 Too Many Requests
- Google Search Central: How HTTP status codes affect Google's crawlers
- Google Search Central: Reduce the Google crawl rate
- Google Search Central: How Google interprets the robots.txt specification
- Google: Debug network and DNS errors for Google's crawlers
- Google Search Central: Managing crawl budget for large sites
- ROAST: Search Central Live Deep Dive Barcelona, Day 1 recap (John Campbell, 30 September 2026)
- Search Engine Roundtable: Google's crawling, indexing and serving timing data (5 October 2026)
Related reading on CrawlRaven
Frequently asked questions
What does the HTTP error code 429 mean?
HTTP 429 means Too Many Requests: the client has sent too many requests in a given amount of time and the server is asking it to slow down, which is commonly called rate limiting. The response may carry a Retry-After header saying how long to wait. It is a 4xx code, so it describes the client's behaviour, not a broken server.
Is a 429 status code bad for SEO?
A short, honest 429 is not harmful, but a sustained or accidental one is. Google's documentation says its crawlers treat 429 as a sign the server is overloaded and slow down crawling, and that the same URL returning 429, 500 or 503 for multiple days may be dropped from the index. The usual damage comes from a WAF serving 429 to Googlebot without anyone noticing.
How does Googlebot handle a 429 compared with other 4xx errors?
Googlebot handles 429 like a server error, not like a 404. Google's status code documentation says all 4xx errors except 429 are treated the same, while 429 is read as an overload signal that prompts crawlers to temporarily slow down. A 404 tells Google the page is gone. A 429 tells Google the page is there but the server cannot cope right now.
How long does a 429 error last?
A 429 lasts as long as the server's rate limit window, which the server can state in a Retry-After header. For Googlebot the after-effect lasts longer than the error. Timings reported from Google's Barcelona event in October 2026 say crawl capacity can drop within seconds, typically takes 4 hours to 1-2 weeks to update, and 1-3 weeks in recovery.
How do I fix a 429 error for Googlebot?
Find what is sending it first. If the origin server is overloaded, add capacity or speed up responses. If a WAF, CDN or bot-protection rule is sending it, change the rule so Google's crawlers are not rate-limited or challenged. Then confirm in the Crawl stats report and your logs that Googlebot is getting 200s again. Google raises the crawl rate on its own once errors fall.
Should I return 429 or 503 to slow Googlebot down?
Either works, and Google's documentation lists 500, 503 and 429 as the codes to return when you urgently need to reduce crawling for a couple of hours or 1 to 2 days. Use 503 for maintenance or an unavailable server and 429 for genuine rate limiting. Google advises against doing it for longer than 1 to 2 days.
Can Cloudflare or another CDN block Googlebot?
Yes, a CDN or WAF can block or rate-limit Googlebot if a rule treats crawl traffic as hostile. Attendees at Search Central Live in Barcelona reported Google saying CDNs are blocking more bot traffic, with a new Cloudflare setup given as an example of something that might end up blocking Googlebot. Those blocks show up as HTTP errors in crawl reporting.
Does a slow time to first byte reduce crawling even without errors?
Yes. Google's crawl budget guide says the crawl capacity limit goes down when a site slows down or responds with server errors, and up when it responds quickly for a while. As reported from Barcelona, connection time and time to first byte sit alongside 429 and 5xx responses as the inputs to the crawl rate limit.
15+ years of growing SaaS websites through SEO | Author, 200-Point Audit Checklist
Aditi has spent 15+ years helping SaaS companies scale organic traffic through technical SEO and content strategy. She is the author of the CrawlRaven 200-Point Audit checklist used by agencies and in-house teams to systematically improve search performance.