lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuaki Hiraga (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3921) Add decompose compound Japanese Katakana token capability to Kuromoji
Date Tue, 27 Mar 2012 06:19:41 GMT


Kazuaki Hiraga commented on LUCENE-3921:

Christian, Thank you for your comments and give us details about your experiments.

I think the length restriction is an acceptable option for splitting the Katakana compound
token.  I tried to use DictionaryCompoundWordTokenFilter for this purpose and we were able
to get the similar result what we expected. But we don't want to rely on another dictionary
that along with dictionaries for Japanese tokenizer.  So, if possible, we would like to have
such functionality in Kuromoji.

> Add decompose compound Japanese Katakana token capability to Kuromoji
> ---------------------------------------------------------------------
>                 Key: LUCENE-3921
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: modules/analysis
>    Affects Versions: 4.0
>         Environment: Cent OS 5, IPA Dictionary, Run with "Search mdoe"
>            Reporter: Kazuaki Hiraga
>              Labels: features
> Japanese morphological analyzer, Kuromoji doesn't have a capability to decompose every
Japanese Katakana compound tokens to sub-tokens. It seems that some Katakana tokens can be
decomposed, but it cannot be applied every Katakana compound tokens. For instance, "トートバッグ(tote
bag)" and "ショルダーバッグ" don't decompose into "トート バッグ" and "ショルダー
バッグ" although the IPA dictionary has "バッグ" in its entry.  I would like to apply
the decompose feature to every Katakana tokens if the sub-tokens are in the dictionary or
add the capability to force apply the decompose feature to every Katakana tokens.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message