robots
Created by Administrator. Last edited by Administrator, 20 years and 125 days ago.
Viewed 3,106 times.
#1
# This file contains User-Agent mappings for robots # # WWW Search Engines Googlebot http://www.googlebot.com/bot.html FAST-WebCrawler http://fast.no/support/crawler.asp Slurp http://www.inktomi.com/slurp.html Teoma http://www.teoma.com/ # # Syndication Crawler Syndic8 http://www.syndic8.com organica http://organica.us Popdexter http://www.popdex.com # # Robots and crawler to be ignored # ZyBorg http://www.WISEnutbot.com IGNORE larbon liston@cc.gatech.edu IGNORE Baiduspider http://www.baidu.com/search/spider.htm IGNORE Infoseek http://www.infoseek.com/ IGNORE TurnitinBot http://www.turnitin.com/robot/crawlerinfo.html IGNORE NIF http://www.newsisfree.com/robot.php IGNORE NPBot http://www.nameprotect.com/botinfo.html IGNORE Robot ??? IGNORE Scooter ??? IGNORE Mercator ??? IGNORE EvilBot ??? IGNORE BlogBot ??? IGNORE
no comments |
post comment
Virtual Dave Megaplex:
Home Page- This wiki's start page
- Send Feedback To Dave
Logged in Users: (0)
Recently Changed
Check Internet Service Database Match- Device-Local Certificate Expired
- CLI Policy Lookup
- DH Selection For IPsec VPNs
- Scan For Newly Added Disk
- nmap
- Seconds Since Epoch to Local Time
- Link Monitor
- DH group to OAKLEY_GROUP table
- Fire Eater
- start
- FortiClient Error Codes
- dhparams Generation
- SSL Certificate Bundles
- ufw
- Grow A LVM Partition
- Creating Users
- whoami
- list databases
- 2023
- 2022
- 2019
- 2018
- 2017
- 2011
- 2010
- 2008
- 2001
- 2000
- 1999
Editing: snipsnap-help, Image Macro(Et auditum est, et idcirco ego nunc simulare)
This is a collection of techical information, much of it learned the hard way. Consider it a lab book or a /info directory. I doubt much of it will be of use to anyone else.