httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Behlendorf <>
Subject Re: Log Ignore Hosts patch uploaded
Date Mon, 26 Feb 1996 23:47:32 GMT
On Mon, 26 Feb 1996, Andrew wrote, quoting Rob Hartill:
> > how about a mod_robots.c which can be made to drop connections from
> > robot sites. Many of them don't deserve a HTTP response, 'cos they
> > ignore its contents anyway. It's much more satisfying if they get
> > absolutely nothing back from sites they attack.
> Talk to the people at Webcrawler and Yahoo, if they don't die
> screaming then build yer module (incorporating mod_hotbabes.c) and
> let's kill this 'brain-dead robot' nonsense off once and for all.

I would really prefer to approach this problem in a manner that doesn't 
conjure up images of "warfare" (maybe I shouldn't have forwarded those 
messages!).  Perhaps some sort of bean-counterish lookup table can be 
done which counts in shared memory a table of hosts and number of hits 
over a given timeframe, and if the number of hits from a particular host is 
over a certain threashold, it drops packets.  I.e., give a 
configuration directive:


which states the maximum number of hits a host can perform, and then once 
every 3 minutes the table is consulted and anyone over a hitrate per 
minute (I'd probably set it around 50 for our pages) doesn't get any 
response from the server for 10 minutes.  There might be some code from 
the acces throttling module someone did.

When HotWired launched in 1994 the early versions of Netscape didn't 
handle authentication quite right - some would send an incorrect 
password, the 401 response would come back, it would resend the bogus 
password, etc., creating a feedback loop.  After launch I would literally 
sit there watching a tail of the logs and editing a .htaccess file in the 
root level of the site everytime I saw an authspam happening.  That was 


--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--  http://www.[hyperreal,organic].com/

View raw message