httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Kallen <>
Subject Re: OT: watch out for robot
Date Sun, 16 Jan 2000 04:30:11 GMT

The other really silly thing about Alexa's bot is its rechecking behavior.
It just can't seem to accept somethings going away and generates repeated
404's.  I configured one my boxes that had been besieged by this to just
403 ia_archiver, haven't analyzed efficacy of this yet though.

Today, Marc Slemko <> frothed and gesticulated about OT:...:
> Just a FYI:'s robot is pathologically stupid.  It pounds on
> servers as hard as it possibly can; over the past few days, it has been
> pounding on one site I've been doing some stuff on at a rate of ~2 million
> hits per day.  I don't know where they get away thinking that sort of
> thing is acceptable, but be warned...
> Obviously, they are no longer welcome at that site.
> User-Agent: ia_archiver
> Coming from: various addresses in

Salon Internet
  Manager, Software and Systems "Livin' La Vida Unix!"
Ian Kallen <> / AIM: iankallen / Fax: (415) 354-3326 

View raw message