infra-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Turner (JIRA)" <>
Subject [jira] Created: (INFRA-1945) Allow crawling of JIRA issues
Date Fri, 20 Mar 2009 02:37:50 GMT
Allow crawling of JIRA issues

                 Key: INFRA-1945
             Project: Infrastructure
          Issue Type: Improvement
      Security Level: public (Regular issues)
            Reporter: Jeff Turner

>From email:

A coworker noticed that one gets crappy results for searches like
because our robots.txt disallows crawling:

$ curl
User-agent: *
Disallow: /jira
Disallow: /scarab
Disallow: /bugzilla
Disallow: /SpamAssassin

Any reason we can't allow crawling of JIRA?  It should be able to handle the load if crawlers
are kept away from saved searches:

Disallow: /jira/sr/
Disallow: /jira/si/
Disallow: /jira/charts


Tony replied:

Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely
caned the hosts they were indexing. I think this is a little less relaxed now, so if no one
has any issues in the next day or so what I suggest you do, is either make the change yourself,
if you can, or open a new JIRA with a patch and we'll do it for you.


This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message