httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tony Rice \(trice\)" <tr...@cisco.com>
Subject [users@httpd] Blocking crawling of CGIs
Date Tue, 18 Sep 2007 18:24:20 GMT
We've had some instances where crawlers have stumbled onto a cgi script
which refers to itself and start pounding the server with requests to
that cgi.

There are so many CGI scripts on this server that I don't want to
maintain a huge robots.txt file.  Any suggestions on other techniques to
keep crawlers away from cgi scripts?  Check the browser with
BrowserMatch and then do something creative with "deny from env="?

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message