HTTP Header Checker

The headers search engines read before they read your page

A crawler acts on several headers before a byte of HTML is parsed, and a few of them can override anything the HTML says. These are the rows this tool pulls out of the raw list and reads for you.

The status code frames all of it: the headers described here belong to the final response, and if that response is a 404 or a 500 the rest of the audit is moot. The chain that led there is shown with each hop's status and timing, because a redirect chain is a finding even when its destination is healthy.

X-Robots-Tag: the noindex you cannot see in the source

Every meta tag checker, and every human doing view-source, audits the HTML. The X-Robots-Tag header carries the same directives one layer down, in the response itself, and that layer wins. A page can declare index, follow in its markup and still be noindexed by one line of server config nobody remembers writing.

• The classic case is a staging rule that shipped. Sites get noindexed at the server level before launch, the launch happens, and the header stays. The HTML was always clean, so nothing in the CMS ever looked wrong.
• It is the only robots control for non-HTML files. A PDF has no head to put a meta tag in, so PDFs, images and feeds are noindexed with this header or not at all.
• It can be scoped too widely. The header is set in server config, and a rule written for one path has a way of matching more than it meant to. One misplaced location block can noindex a directory.
• The more restrictive signal wins. When header and meta tag disagree, Google honours the stricter one, in either direction. Keeping both is keeping two things that can drift.

Caching headers are crawl budget wearing a different name

Caching headers read like a performance topic, and they are, but the crawler consequence is the one this tool flags. When a response carries an ETag or Last-Modified, a returning crawler can ask the server whether the page changed and receive a 304 Not Modified, a response of a few hundred bytes instead of the whole page. Google has said plainly that it crawls sites supporting conditional requests more efficiently.

Without a validator, every recrawl of every URL is a full download. One page does not notice; a site with fifty thousand URLs is spending its crawl budget re-transferring content that has not changed since the last visit. The same absence also means every CDN and proxy between you and a visitor is guessing at a policy you never set, which is a page speed cost on top of the crawl one.

Compression is the third leg of the same audit. A missing Content-Encoding on an HTML response means the page travels at full size on every request, and the fix is one switch in the same config file the caching headers live in. This tool requests with gzip and brotli accepted, so what it reports is what the server chose to do with a client that supports everything.

How to check a URL's HTTP headers

Seven steps, ordered by how expensive the finding is when it is missed.

1. 01

   Check the URL exactly as it is linked

   Headers can differ between the www and bare domain, between http and https, and between trailing-slash variants, because those are different URLs that may hit different server rules. Check the spelling that is actually in use.

2. 02

   Read X-Robots-Tag before anything else

   It is the one header that can silently remove a page from Google, it overrides the meta robots tag, and no view-source will ever show it. If a page has vanished from the index and its HTML looks clean, this header is the first suspect.

3. 03

   Walk the redirect chain hop by hop

   The headers you get belong to the final URL, but the hops before it each cost latency and crawl budget. More than one redirect, or a downgrade from HTTPS to HTTP anywhere in the chain, is a finding of its own.

4. 04

   Look for a canonical in the Link header

   Google reads a Link header canonical exactly like the tag in the head. When the two disagree, or when the header carries two canonicals, Google ignores both and chooses for itself.

5. 05

   Check the caching pair: a validator and a policy

   An ETag or Last-Modified lets the server answer recrawls with 304 Not Modified instead of the full page, and Cache-Control tells every cache in between what it may keep. A page with neither is re-downloaded in full on every visit and every crawl.

6. 06

   Confirm the response is compressed

   A Content-Encoding of gzip, br or zstd should appear on any HTML response. Its absence on a full page usually means a config gap that is costing transfer time on every single request.

7. 07

   Fix the security headers in the same visit

   None of them are ranking factors, but header work happens in one config file, and adding HSTS, nosniff and a frame policy while you are already in it is nearly free.

The neighbouring tools each take one of these threads further: the redirect chain checker focuses entirely on hops and loop detection, the robots.txt tester covers the other way crawlers get blocked, and the canonical tag checker compares the HTML canonical against what this tool finds in the Link header.

What this checker reports

Frequently asked questions