lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Ingersoll" <>
Subject Tokenizers and java.text.BreakIterator
Date Tue, 20 Jul 2004 19:23:33 GMT

Was wondering if anyone uses java.text.BreakIterator#getWordInstance(Locale) as a tokenizer
for various languages?  Does it do a good job?  It seems like it does, at least for languages
where words are separated by spaces or punctuation, but I have only done simple tests.

Anyone have any thoughts on this?  What am I missing?  Does this seem like a valid approach?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message