lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <>
Subject Re: [contrib]: StandardTokenizer with sigram based CJK Support
Date Tue, 27 Aug 2002 16:26:11 GMT

Che Dong wrote:
>>Attached  StandardTokenizer.jj with Sigram Based east
>>asia language support:
>>tested under Windows and GNU/Linux
>>Just treat different UnicodeBlock with different word
>>segment method. 
>>Hope in the future released we can add more language
>>support in StandardTokenizer.jj step by step and keep
>>it fit for most i18n environment.
>>Some common app, like Jive, can use it as default
>>Use localized Analyzier for advanced usage.
>>Thank you.
>>Che, Dong

To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

View raw message