httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bgo...@crewcorpinc.com
Subject Re: [users@httpd] Can I block.crawlers from seeing files with certain suffixes, i.e. .php? Thanks.
Date Fri, 12 Jul 2013 16:04:34 GMT
An ideal way to do so would be to check the user agent of the crawler in your index page and
then link to static content that you want the crawler to see. Minimizes cpu cycles as well
as reduces the error they are seeing.
------Original Message------
From: mrl
To: users@httpd.apache.org
ReplyTo: users@httpd.apache.org
Subject: [users@httpd] Can I block.crawlers from seeing files with certain suffixes, i.e.
.php?  Thanks.
Sent: Jul 12, 2013 11:34 AM

Is there a way to block .php from being indexed by crawlers, but allow other 
type files to be indexed?  When the crawlers access the php files, they are 
executed, creating lots of error messages (and taking up cpu cycles).  Thanks. 
- Mark 



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Sent from my BlackBerry device on the Rogers Wireless Network

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message