lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Moen (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3767) Explore streaming Viterbi search in Kuromoji
Date Thu, 23 Feb 2012 02:29:48 GMT


Christian Moen commented on LUCENE-3767:

Robert, some comments are below.

And I tend to like Mike's improvements from a relevance perspective for these reasons:

# keeping the original compound term for improved precision
# preventing compound decomposition from having any unrelated negative impact on the rest
of the tokenization

Very good points.

So I think we should pursue this change, even if we want to separately train a dictionary
in the future, because in that case, we would just disable the kanji decomposition heuristic
but keep the heuristic (obviously re-tuned!) for katakana?

I agree completely.

The dictionary documentation for the original ipadic has the ability to hold compound data
(not in mecab-ipadic though, so maybe it was never implemented?!), but I don't actually see
it in any implementations. So yeah, we would need to find a corpus containing compound information
(and of course extend the file format and add support to kuromoji) to support that.

However, would this really solve the total issue? Wouldn't that really only help for known
kanji compounds... whereas most katakana compounds (e.g. the software engineer example) are
expected to be OOV anyway? So it seems like, even if we ensured the dictionary was annotated
for long kanji such that we always used decompounded forms, we need a 'heuristical' decomposition
like search-mode either way, at least for the unknown katakana case?

I've made an inquiry to a friend who did his PhD work at Prof. Matsumoto's lab at NAIST (where
ChaSen was made) and I've made en inquiry regarding compound information and the Kyoto Corpus.

You are perfectly right that this doesn't solve the complete problem as unknown words can
actually be compounds -- unknown compounds.  The approach used today is basically adding all
the potential decompounds the model knows about to the lattice and see if a short path can
be found often in combination with an unknown word.

We get errors such as クイーンズコミックス (Queen's Comics) becoming クイーン ズコミックス
(Queen Zukuomikkusu) because クイーン (Queen) is known.

I'll open up a separate JIRA for discussing search-mode improvements. :)

> Explore streaming Viterbi search in Kuromoji
> --------------------------------------------
>                 Key: LUCENE-3767
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: modules/analysis
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.6, 4.0
>         Attachments: LUCENE-3767.patch, LUCENE-3767.patch, LUCENE-3767.patch, compound_diffs.txt
> I've been playing with the idea of changing the Kuromoji viterbi
> search to be 2 passes (intersect, backtrace) instead of 4 passes
> (break into sentences, intersect, score, backtrace)... this is very
> much a work in progress, so I'm just getting my current state up.
> It's got tons of nocommits, doesn't properly handle the user dict nor
> extended modes yet, etc.
> One thing I'm playing with is to add a double backtrace for the long
> compound tokens, ie, instead of penalizing these tokens so that
> shorter tokens are picked, leave the scores unchanged but on backtrace
> take that penalty and use it as a threshold for a 2nd best
> segmentation...

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message