httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: [users@httpd] 404's to robots.txt?
Date Wed, 22 Jul 2009 00:03:54 GMT
Evan Platt wrote:
> So I've noticed quite a lot of connections from web spider programs. 
> I've had a robots.txt
> (User-agent: *
> Disallow: /)  For a long time. But looking closer in my apache logs, am 
> I reading right that it's giving a 404?
> 
> 65.55.106.173 - - [21/Jul/2009:09:44:43 -0700] "GET /robots.txt 
> HTTP/1.1" 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.112 - - [21/Jul/2009:10:11:43 -0700] "GET /robots.txt 
> HTTP/1.1" 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.166 - - [21/Jul/2009:11:03:35 -0700] "GET /robots.txt 
> HTTP/1.1" 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.160 - - [21/Jul/2009:11:09:07 -0700] "GET /robots.txt 
> HTTP/1.1" 200 28 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.180 - - [21/Jul/2009:11:35:34 -0700] "GET /robots.txt 
> HTTP/1.1" 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 
> Same day, no changes made:
> X.X.X.X - - [21/Jul/2009:16:47:44 -0700] "GET /robots.txt HTTP/1.1" 304 
> - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1) 
> Gecko/20090715 Firefox/3.0.7, Ant.com Toolbar 1.3 (.NET CLR 3.5.30729)"
> Z.Z.Z.Z- - [21/Jul/2009:16:49:10 -0700] "GET /robots.txt HTTP/1.1" 200 
> 28 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) 
> AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.30 Safari/530.5"
> 
> Two different IP's. One myne, one a friends.
> 
> Any suggestions as to why (if I'm reading the log right) I'm handing out 
> a 404 to it appears just web crawlers?
> 
> # httpd -v
> Server version: Apache/2.2.3
> Server built:   Jun 16 2009 11:28:50
> 
> Don't know what other information is needed to help troubleshoot... 
> Running on a os//x box.
> http://www.espphotography.com/robots.txt if you want to take a look...
> 
I had a look, can access it also.
Maybe you should look in your error.log, to check /why/ it refuses the 
other ones ?
Maybe there is another rule in your config that disallows that other 
client's address, or user-agent ?

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message