httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Sutton <p...@awe.com>
Subject killing robots
Date Mon, 09 Feb 1998 14:17:29 GMT
Umm, www.apacheweek.com is being attacked by a nasty robot. None of the
other vhosts we have are affected though. Perhaps it doesn't like apache?
Just thought I'd let you know in case it is attacking other apache-related
sites. 

We got 170,000 hits from it last week (fairly noticeable since we normally
only get 40,000 or so). It is coming from 193.136.17.202
(donald.di.uminho.pt) with a UA of "GETWWW-ROBOT/2.0".

We are also getting a few hits from another robot-like thing: from
118.40.17.203 (dp-m-a18.werple.net.au) with UA "Java1.1.3" (there is also
a Java1.1.4 agent out there, but that has only made a few requests). The
robot seems particularly broken -- we use multiviews on every request, but
Java1.1.3 seems to always add a trailing / unless the link contained an
extension, then it tries without the /.

Anyway, what's the current wisdom on how to deal with robots? Do you match
its UA & IP, then reject with a 404 or 500, or just trash the whole IP?  I
haven't really kept up with the robot wars, so any advice would be useful. 
Is there a good site which tracks nasty robot issues?

Paul


Mime
View raw message