roller-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kohei Nozaki (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ROL-2090) Lucene integration doesn't work well for entries that written in some languages
Date Sun, 06 Dec 2015 03:29:11 GMT
Kohei Nozaki created ROL-2090:
---------------------------------

             Summary: Lucene integration doesn't work well for entries that written in some
languages
                 Key: ROL-2090
                 URL: https://issues.apache.org/jira/browse/ROL-2090
             Project: Apache Roller
          Issue Type: Improvement
          Components: Data Model & JPA Backend
    Affects Versions: 5.1.2
            Reporter: Kohei Nozaki
            Assignee: Roller Unassigned
            Priority: Minor


Reported in http://benzaiten.dyndns.org/roller/ugya/entry/roller_500_to_510_migration (Japanese).
Summary in English:

h4. Japanese keywords doesn't hit against the latter part of long entry

It's caused by maximum token limit in the following code. The author said that typical Japanese
text is not splitted by white spaces so that's not work well with it.

{noformat}
// Limit to 1000 tokens.
LimitTokenCountAnalyzer analyzer = new LimitTokenCountAnalyzer(
        IndexManagerImpl.getAnalyzer(), 1000);
{noformat}

h4. StandardAnalyzer doesn't work well with Japanese text

Roller uses {{StandardAnalyzer}} but there are some other language specific implementations
for it such as {{CJKAnalyzer}} or {{JapaneseAnalyzer}}. The author said that these implementations
improve accuracy for such languages. I know these implementations are language specific so
we can't simply replace it to them but might want to switch it in flexible manner, Such as
using language configuration in each blogs.

{noformat}
public static final Analyzer getAnalyzer() {
    return new StandardAnalyzer(FieldConstants.LUCENE_VERSION);
}
{noformat}

I'm still not sure what would be proper solutions but I believe we have room for some improvement
here. Any advices would be appreciated.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message