httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-liang Kao <cl...@pamud.net>
Subject Re: killing robots
Date Mon, 09 Feb 1998 16:23:38 GMT
on 02/09/98 Mon, Paul Sutton <paul@awe.com> wrote:
> Anyway, what's the current wisdom on how to deal with robots? Do you match
> its UA & IP, then reject with a 404 or 500, or just trash the whole IP?  I
> haven't really kept up with the robot wars, so any advice would be useful. 
> Is there a good site which tracks nasty robot issues?
>
> Paul
There is a `Standard for Robot Exclusion' indicates the robots should not
grab data from sites with /robots.txt which specified content.

refer to: http://info.webcrawler.com/mak/projects/robots/exclusion.html

But bad-mannered robots can simply not implement to obey the standards.

Just my $0.02. 

CLK
-- 
Chia-liang Kao  /  clkao@cirx.org
Panther Tech Co. , Taichung, Taiwan
http://www.pamud.net/~clkao
`白爛濤濤我不怕' -- IOI 97

Mime
View raw message