infra-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Aizatsky (JIRA)" <j...@apache.org>
Subject [jira] Created: (INFRA-1578) Allow GoogleCodeBot in robots.txt
Date Mon, 07 Apr 2008 11:26:24 GMT
Allow GoogleCodeBot in robots.txt
---------------------------------

                 Key: INFRA-1578
                 URL: https://issues.apache.org/jira/browse/INFRA-1578
             Project: Infrastructure
          Issue Type: Wish
      Security Level: public (Regular issues)
          Components: Website
            Reporter: Mike Aizatsky


Hello,

We, at google, has received quite a few complaints about Apache
software source code being unavailable on Google Code Search
(http://www.google.com/codesearch). We've investigated the issue, and
found that you have a robots.txt file disallowing even our special
google code crawlers (http://svn.apache.org/robots.txt):

User-agent: *
Disallow: /

We do believe this was done to tell usual web crawlers to stay away
from your svn repositories, but we have a custom,
svn-interface-conformant crawler in codesearch. Can you relax your
robots.txt for us and allow "GoogleCodeBot" to index your site? Or if
you're reluctant to change your file, can you just confirm that we're
free to index your source code?

--
Regards,
Mike

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message