>
> > Well the robot folks seem to have reluctanly agreed to add "/robot"
> > to USER-AGENT so that servers can react to it.
>
> Not a good idea -- it won't last more than three months before idiots
> start changing content based on that, at which time the typical index
> robots will just have to mimic Lynx or Netscape. In any case, "/robot"
> is not valid HTTP syntax.
it looks like the idiots are already using different techniques to
achieve this and by now all the popular search phrases are already
useless. If people want to spoof the robots they'll continue to find ways to
do it with or without help form a robot identifier.
So if "/robot" is bad (Koster seems to think it's ok), then what would
be valid?
rob
|