Return-Path: Delivered-To: new-httpd-archive@hyperreal.org Received: (qmail 21850 invoked by uid 6000); 9 Feb 1998 16:23:32 -0000 Received: (qmail 21842 invoked from network); 9 Feb 1998 16:23:21 -0000 Received: from ns1.tc.neto.net (HELO ns1.pamud.net) (clkao@210.209.60.1) by taz.hyperreal.org with SMTP; 9 Feb 1998 16:23:21 -0000 Received: (from clkao@localhost) by ns1.pamud.net (8.8.8/8.8.8) id AAA21905; Tue, 10 Feb 1998 00:23:38 +0800 (CST) Message-ID: <19980210002338.46097@pamud.net> Date: Tue, 10 Feb 1998 00:23:38 +0800 From: Chia-liang Kao To: new-httpd@apache.org Subject: Re: killing robots References: Mime-Version: 1.0 Content-Type: text/plain; charset=big5 Content-Transfer-Encoding: 8bit X-Mailer: Mutt 0.81 In-Reply-To: ; from Paul Sutton on Mon, Feb 09, 1998 at 02:17:29PM +0000 Sender: new-httpd-owner@apache.org Precedence: bulk Reply-To: new-httpd@apache.org on 02/09/98 Mon, Paul Sutton wrote: > Anyway, what's the current wisdom on how to deal with robots? Do you match > its UA & IP, then reject with a 404 or 500, or just trash the whole IP? I > haven't really kept up with the robot wars, so any advice would be useful. > Is there a good site which tracks nasty robot issues? > > Paul There is a `Standard for Robot Exclusion' indicates the robots should not grab data from sites with /robots.txt which specified content. refer to: http://info.webcrawler.com/mak/projects/robots/exclusion.html But bad-mannered robots can simply not implement to obey the standards. Just my $0.02. CLK -- Chia-liang Kao / clkao@cirx.org Panther Tech Co. , Taichung, Taiwan http://www.pamud.net/~clkao `�������ܧڤ���' -- IOI 97