lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Updated] (LUCENE-2522) add simple japanese tokenizer, based on tinysegmenter
Date Sat, 23 Apr 2011 23:29:06 GMT


Robert Muir updated LUCENE-2522:

    Attachment: LUCENE-2522.patch

attached is an updated patch, its still a work in progress (needs some more tests and benchmarking
and some other things little fixes).

Theres a general pattern for these segmenters (this one, smartchinese, sen) thats a little
tricky, that is they want to really look at sentences to determine how to segment.

So, I added a base class for this to make writing these segmenters easier, and also to hopefully
improve segmentation accuracy. (I would like to switch smartchinese over to it) This class
makes it easy to segment sentences with a Sentence BreakIterator... in my opinion it doesnt
matter how theoretically good the word tokenization is for these things, if the sentence tokenizer
is really bad (I found this issue with both sen and smartchinese).

hope to get it committable soon

> add simple japanese tokenizer, based on tinysegmenter
> -----------------------------------------------------
>                 Key: LUCENE-2522
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/analyzers
>            Reporter: Robert Muir
>            Priority: Minor
>             Fix For: 4.0
>         Attachments: LUCENE-2522.patch, LUCENE-2522.patch, LUCENE-2522.patch
> TinySegmenter ( is a tiny japanese
> It was ported to java/lucene by Kohei TAKETA <>, 
> and is under friendly license terms (BSD, some files explicitly disclaim copyright to
the source code, giving a blessing instead)
> Koji knows the author, and already contacted about incorporating into lucene:
> {noformat}
> I've contacted Takeda-san who is the creater of Java version of
> TinySegmenter. He said he is happy if his program is part of Lucene.
> He is a co-author of my book about Solr published in Japan, BTW. ;-)
> {noformat}

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message