The following is a basic listing of search engine spider names and their "owners". This is by no means complete, as there are many thousands of search engines on the Internet, but it covers the more common beneficial spiders. Look for these in your traffic reports or search for the names through your server logs to discover which pages they have been spidering. You'll find that many of the entries will also have accompanying numbers or letters e.g Googlebot/2.1 or Slurp.so/1.0
Spider name | Spider owner |
| Googlebot | Google.com |
| MSNbot | Search.msn.com |
| Ask Jeeves/Teoma | Ask.com |
| Architext spider | Excite.com |
| FAST-WebCrawler | AllTheWeb.com |
| Slurp | Inktomi.com |
| Yahoo Slurp | Yahoo Web Search |
| ia_archiver | Alexa.com |
| Scooter | AltaVista.com |
| crawler@fast | AllTheWeb.com |
| Crawler | Crawler.de |
| InfoSeek sidewinder | InfoSeek.com |
| Lycos_Spider_(T-Rex) | Lycos.com |
If you have spotted any significant activity from these spiders in your reports or logs, there's a good chance that you'll be listed on that particular search engine. But you'll need to be patient; some Search Engines take far longer than others to refresh their databases!
0 comments:
Post a Comment