httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Covener <cove...@gmail.com>
Subject Re: [users@httpd] 404's to robots.txt?
Date Wed, 22 Jul 2009 01:03:47 GMT
On Tue, Jul 21, 2009 at 7:56 PM, Evan Platt<evan@espphotography.com> wrote:
> So I've noticed quite a lot of connections from web spider programs. I've
> had a robots.txt
> (User-agent: *
> Disallow: /)  For a long time. But looking closer in my apache logs, am I
> reading right that it's giving a 404?
>
> 65.55.106.173 - - [21/Jul/2009:09:44:43 -0700] "GET /robots.txt HTTP/1.1"
> 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.112 - - [21/Jul/2009:10:11:43 -0700] "GET /robots.txt HTTP/1.1"
> 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.166 - - [21/Jul/2009:11:03:35 -0700] "GET /robots.txt HTTP/1.1"
> 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.160 - - [21/Jul/2009:11:09:07 -0700] "GET /robots.txt HTTP/1.1"
> 200 28 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
> 65.55.106.180 - - [21/Jul/2009:11:35:34 -0700] "GET /robots.txt HTTP/1.1"
> 404 208 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
>


More than 1 docroot / log %{Host}i ?

-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message