www-infrastructure-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "#asfinfra IRC Bot (JIRA)" <j...@apache.org>
Subject [jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
Date Fri, 20 Mar 2009 02:45:50 GMT

    [ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683728#action_12683728
] 

#asfinfra IRC Bot commented on INFRA-1945:
------------------------------------------

<pquerna> also, we should let google hit bugzilla.


> Allow crawling of JIRA issues
> -----------------------------
>
>                 Key: INFRA-1945
>                 URL: https://issues.apache.org/jira/browse/INFRA-1945
>             Project: Infrastructure
>          Issue Type: Improvement
>      Security Level: public(Regular issues) 
>            Reporter: Jeff Turner
>            Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66
because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA?  It should be able to handle the load if
crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely
caned the hosts they were indexing. I think this is a little less relaxed now, so if no one
has any issues in the next day or so what I suggest you do, is either make the change yourself,
if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message