lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Ingersoll" <gsing...@syr.edu>
Subject Tokenizers and java.text.BreakIterator
Date Tue, 20 Jul 2004 19:23:33 GMT
Hi,

Was wondering if anyone uses java.text.BreakIterator#getWordInstance(Locale) as a tokenizer
for various languages?  Does it do a good job?  It seems like it does, at least for languages
where words are separated by spaces or punctuation, but I have only done simple tests.

Anyone have any thoughts on this?  What am I missing?  Does this seem like a valid approach?

Thanks,
Grant


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message