httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-liang Kao <>
Subject Re: killing robots
Date Mon, 09 Feb 1998 16:23:38 GMT
on 02/09/98 Mon, Paul Sutton <> wrote:
> Anyway, what's the current wisdom on how to deal with robots? Do you match
> its UA & IP, then reject with a 404 or 500, or just trash the whole IP?  I
> haven't really kept up with the robot wars, so any advice would be useful. 
> Is there a good site which tracks nasty robot issues?
> Paul
There is a `Standard for Robot Exclusion' indicates the robots should not
grab data from sites with /robots.txt which specified content.

refer to:

But bad-mannered robots can simply not implement to obey the standards.

Just my $0.02. 

Chia-liang Kao  /
Panther Tech Co. , Taichung, Taiwan
`白爛濤濤我不怕' -- IOI 97

View raw message