Add a page to AskRobots

Submit a public page. We fetch it right away and it is usually in search results within a minute.

Not sure it's needed? Check shows what we already know about the address (in the index, when we last read it, when it refreshes) without fetching anything.

What happens

We check the address, then fetch the page the way our crawler always does: as AskRobotsBot, following your robots.txt, and without fetching from your site faster than our usual pace. A page that passes the same quality checks as every crawled page goes into the index, and you see it the moment it is searchable.

After that, it is part of our normal crawl: we revisit it now and then to keep it fresh.

What it doesn't do

Submitting a page gets it seen; it never moves it up in results. Pages are ranked the same way whether they were submitted or found by the crawler.

We don't fetch pages that robots.txt keeps us out of, sites that answer with a bot challenge, private or local addresses, or sites on our adult, malware or phishing lists.

For scripts

curl -X POST -H 'Content-Type: application/json' \
     -d '{"url": "https://example.com/page"}' \
     https://search.askrobots.com/api/submit

The answer has an id and a status_api address to poll. The same limits apply.