httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Hartill <>
Subject Re: robot denial
Date Fri, 19 Jul 1996 10:34:07 GMT
> > Well the robot folks seem to have reluctanly agreed to add "/robot"
> > to USER-AGENT so that servers can react to it.
> Not a good idea -- it won't last more than three months before idiots
> start changing content based on that, at which time the typical index
> robots will just have to mimic Lynx or Netscape.  In any case, "/robot"
> is not valid HTTP syntax.

it looks like the idiots are already using different techniques to 
achieve this and by now all the popular search phrases are already
useless. If people want to spoof the robots they'll continue to find ways to
do it with or without help form a robot identifier.

So if "/robot" is bad (Koster seems to think it's ok), then what would
be valid?


View raw message