httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tony Rice \(trice\)" <>
Subject [users@httpd] Blocking crawling of CGIs
Date Tue, 18 Sep 2007 18:24:20 GMT
We've had some instances where crawlers have stumbled onto a cgi script
which refers to itself and start pounding the server with requests to
that cgi.

There are so many CGI scripts on this server that I don't want to
maintain a huge robots.txt file.  Any suggestions on other techniques to
keep crawlers away from cgi scripts?  Check the browser with
BrowserMatch and then do something creative with "deny from env="?

The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:> for more info.
To unsubscribe, e-mail:
   "   from the digest:
For additional commands, e-mail:

View raw message