About this page
Robots Exclusion
https://web.archive.org/web/20071107021800/http://www.robotstxt.org/wc/exclusion.html
“Sometimes people find they have been indexed by an indexing robot, or that a resource discovery robot has visited part of a site that for some reason shouldn't be visited by robots. In recognition of this problem, many Web Robots offer facilities for Web site administrators and content providers to...” (from the page’s text)
- Topic
- Web Development
- Quality
- Quality 84 scored 12 Aug 2025
- Language
- english
- Text on page
- 2,044 characters
- Page size
- Not measured yet
- Answered
- Not fetched yet
- Last read
- 6 Jul 2025
- In our index since
- 6 Jul 2025
- Safe search
- Not checked yet
- Links to it
- Linked from 1 site