lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Smith" <>
Subject RE: CJK Analyzer indexing japanese word document
Date Tue, 16 Mar 2004 22:42:46 GMT
I have used this analyzer with Japanese and it works fine.  In fact, I'm
currently doing English, several western European languages, traditional
and simplified Chinese and Japanese.  I throw them all in the same index
and have had no problem other than my users wanted the search limited by
language.  I solved that problem by simply adding a keyword field to the
Document which has the 2-letter language code.  I then automatically add
the term indicating the language as an additional constraint when the
user specifies the search.  

You do need to be sure that the Shift-JIS gets converted to unicode
before you put it in the Document (and pass it to the analyzer).
Internally, I believe lucene wants everything in unicode (as any good
java program would). Originally, I had problems with Asian languages and
eventually determined my xml parser wasn't translating my Shift-JIS,
Big5, etc. to unicode.  Once I fixed that, life was good.

-----Original Message-----
From: Che Dong [] 
Sent: Tuesday, March 16, 2004 8:31 AM
To: Lucene Users List
Subject: Re: CJK Analyzer indexing japanese word document

some Korean friends tell me they use it successfully for Korean. So I
think its also work for Japanese. mostly the problem is locale settings

Please check weblucene project for xml indexing samples: 

Che Dong
----- Original Message -----
From: "Chandan Tamrakar" <>
To: <>
Sent: Tuesday, March 16, 2004 4:31 PM
Subject: CJK Analyzer indexing japanese word document

> I am using a CJKAnalyzer from apache sandbox , I have set the java
> file.encoding setting to SJIS
> and  i am able to index and search the japanese html page . I can see
> index dumps as i expected , However when i index a word document
> japanese characters it is not indexing as expected . Do I need to
> anything with CJKTokenizer and CJKAnalyzer classes?
> I have been able to index a word document with StandardAnalyzers.
> thanks in advace
> chandan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message