About the AskRobots crawler

AskRobots is an independent, honest web search index. AskRobotsBot is the robot that reads public pages for it. This page is for people who run websites.

What it does

AskRobotsBot fetches public web pages, reads their text and links, and adds them to the AskRobots search index. It does not log in, fill in forms, run scripts or fetch pages that robots.txt disallows. Pages it indexes can appear in results at search.askrobots.com, with a link back to your site.

How to recognise it

Its User-Agent header is:

AskRobotsBot/1.0 (+https://search.askrobots.com/bot)

It crawls only from this IP address, so you can check a suspicious request against it:

162.224.53.162

A request that claims to be AskRobotsBot from any other address is not us.

How often it visits

It waits at least one second between requests to the same site, and longer if your robots.txt asks for it. It fetches your robots.txt before crawling and refreshes it about once a day. If robots.txt answers with a server error (5xx or 429), it stops crawling that site for an hour and tries again. Pages that return an error or no longer exist are revisited rarely.

Controlling it with robots.txt

AskRobotsBot follows the Robots Exclusion Protocol. Use the token AskRobotsBot after User-agent:. Rules for * apply to it too, unless a group names AskRobotsBot.

Block it completely

User-agent: AskRobotsBot
Disallow: /

Slow it down

User-agent: AskRobotsBot
Crawl-delay: 10

Crawl-delay is the number of seconds to wait between requests.

Keep it out of some paths

User-agent: AskRobotsBot
Disallow: /private/
Disallow: /search

Changes are picked up within a day.

Removal requests and problems

To have pages removed from the index, report abusive crawling, or tell us something is wrong, write to crawler@askrobots.com. Include your domain and, for abuse reports, a few log lines with timestamps. Blocking AskRobotsBot in robots.txt always works and needs no contact with us.