> I have a server farm dishing out about 100 domains' webpages using
> LAMP. Have you noticed since Google announce gmail, the other search
> engine bots (in particular msnbot and yahoo slurp), have become
> inordinately intrusive? They obey robots.txt, but they now hit sites as
> if they were visiting as opposed to just gathering relevant info e.g.
> using ?=phpsession yada yada etc.
I find msnbot to be worst; thousands of hits per day on each website,
day after day after day. You don't need to do that just for indexing.
Furthermore, we're between a rock and a hard place--if you tell
msnbot to take a hike (via robots.txt), then you may hurt yourself wrt
SE placement on msn.
John
-----------------------------------------------------------------------
This list is provided as an unmoderated internet service by Networked
Knowledge Systems (NKS). Views and opinions expressed in messages
posted are those of the author and do not necessarily reflect the
official policy or position of NKS or any of its employees.
This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 17:43:56 EDT