www-infrastructure-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Turner (JIRA)" <j...@apache.org>
Subject [jira] Created: (INFRA-1945) Allow crawling of JIRA issues
Date Fri, 20 Mar 2009 02:37:50 GMT
Allow crawling of JIRA issues
-----------------------------

                 Key: INFRA-1945
                 URL: https://issues.apache.org/jira/browse/INFRA-1945
             Project: Infrastructure
          Issue Type: Improvement
      Security Level: public (Regular issues)
            Reporter: Jeff Turner


>From email:

A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66
because our robots.txt disallows crawling:

$ curl http://issues.apache.org/robots.txt
User-agent: *
Disallow: /jira
Disallow: /scarab
Disallow: /bugzilla
Disallow: /SpamAssassin

Any reason we can't allow crawling of JIRA?  It should be able to handle the load if crawlers
are kept away from saved searches:

Disallow: /jira/sr/
Disallow: /jira/si/
Disallow: /jira/charts

--Jeff

Tony replied:


Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely
caned the hosts they were indexing. I think this is a little less relaxed now, so if no one
has any issues in the next day or so what I suggest you do, is either make the change yourself,
if you can, or open a new JIRA with a patch and we'll do it for you.


Cheers,
Tony

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message