httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph S D Yao <j...@tux.org>
Subject Re: [users@httpd] Stoping bots from sucking files
Date Fri, 24 Oct 2008 19:54:27 GMT
On Fri, Oct 24, 2008 at 08:29:43PM +0100, Mário Gamito wrote:
> Hi,
> 
> I have this site that has a directory with some files.
> A few weeks ago, two web bots started sucking those files at an impressive rate.


Use a robots.txt file in your home directory.

http://www.robotstxt.org/
http://en.wikipedia.org/wiki/Robots.txt

If they ignore it, then use iptables to block them.  That takes the
strain off httpd.


-- 
/*********************************************************************\
**
** Joe Yao				jsdy@tux.org - Joseph S. D. Yao
**
\*********************************************************************/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message