incubator-ooo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject DO NOT REPLY [Bug 118940] New: Tune robots.txt for website
Date Mon, 20 Feb 2012 23:06:59 GMT

             Bug #: 118940
        Issue Type: DEFECT
           Summary: Tune robots.txt for website
    Classification: Infrastructure
           Product: www
           Version: version58
          Platform: PC
        OS/Version: Linux
            Status: CONFIRMED
          Severity: normal
          Priority: P3
         Component: website general issues

We currently have:

User-agent: daumoa
Disallow: /

User-Agent: Sosospider
Disallow: /

User-Agent: Snoopy
Disallow: /

User-Agent: *
Crawl-delay: 5
Disallow: /nonav/
Disallow: /welcome/
Disallow: /white_papers/

Some of this doesn't make sense.  For example, there is nothing at /nonav,
/welcome seems to have a redirect issue that we should just fix.  And
/white-papers/ has reasonable content that we should not be hiding from index

Configure bugmail:
------- You are receiving this mail because: -------
You are on the CC list for the bug.

View raw message