RavenCrawl

How to recognize it

Every request sends this user agent:

RavenCrawl/0.6.0 (+https://crawlraven.com/bot)

The version number changes with releases; the RavenCrawl name and this page's address stay the same. We don't publish IP addresses for RavenCrawl, and any client can send this user agent, so treat it as a label rather than proof of who is asking.

What it requests

• Pages of the site being audited: its robots.txt and sitemap, then the pages it reaches from there, following robots.txt.
• Link and image checks: for links and images on those pages, including ones on other sites, a HEAD request to see whether the URL still works, followed by a GET when the HEAD request returns an error. These single checks don't read the linked site's robots.txt.

RavenCrawl doesn't sign in, submit forms, or try to get past CAPTCHAs or bot challenges.

Blocking it with robots.txt

RavenCrawl follows the robots.txt group that names RavenCrawl, or the * group when none does. To keep it from crawling your site in audits:

User-agent: RavenCrawl
Disallow: /

• If your robots.txt returns a server error or can't be reached, RavenCrawl doesn't crawl the site.
• Crawl-delay isn't read.
• robots.txt covers audit crawls of your site. A page elsewhere that links to yours can still lead to a single link check of that URL; to stop those, contact us.

Where it runs

Audits run by CrawlRaven always send the user agent above and follow robots.txt. RavenCrawl is also built into tools people run on their own computers. Those send the same user agent and follow robots.txt by default, but the person running them can change both, and their requests come from their own network.

Contact

To report a problem, or to ask us to stop requests to your site, email hello@crawlraven.com or use the contact page. Include the time of the requests, the URLs requested, and the user agent you saw.