httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Kallen <spida...@salon.com>
Subject Re: OT: watch out for alexa.com robot
Date Sun, 16 Jan 2000 04:30:11 GMT

The other really silly thing about Alexa's bot is its rechecking behavior.
It just can't seem to accept somethings going away and generates repeated
404's.  I configured one my boxes that had been besieged by this to just
403 ia_archiver, haven't analyzed efficacy of this yet though.

Today, Marc Slemko <marcs@znep.com> frothed and gesticulated about OT:...:
> Just a FYI: alexa.com's robot is pathologically stupid.  It pounds on
> servers as hard as it possibly can; over the past few days, it has been
> pounding on one site I've been doing some stuff on at a rate of ~2 million
> hits per day.  I don't know where they get away thinking that sort of
> thing is acceptable, but be warned...
> 
> Obviously, they are no longer welcome at that site.
> 
> User-Agent: ia_archiver
> Coming from: various addresses in 206.132.186.0/24

--
Salon Internet 				http://www.salon.com/
  Manager, Software and Systems "Livin' La Vida Unix!"
Ian Kallen <idk@salon.com> / AIM: iankallen / Fax: (415) 354-3326 


Mime
View raw message