infra-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "#asfinfra IRC Bot (JIRA)" <>
Subject [jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
Date Fri, 20 Mar 2009 03:21:50 GMT


#asfinfra IRC Bot commented on INFRA-1945:

<jefft> Made the suggested change for all the JIRA prefixes

> Allow crawling of JIRA issues
> -----------------------------
>                 Key: INFRA-1945
>                 URL:
>             Project: Infrastructure
>          Issue Type: Improvement
>      Security Level: public(Regular issues) 
>            Reporter: Jeff Turner
>            Assignee: Jeff Turner
> From email:
> A coworker noticed that one gets crappy results for searches like
because our robots.txt disallows crawling:
> $ curl
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA?  It should be able to handle the load if
crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely
caned the hosts they were indexing. I think this is a little less relaxed now, so if no one
has any issues in the next day or so what I suggest you do, is either make the change yourself,
if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message