Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Users List" <lucene-user@jakarta.apache.org>
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Does Lucene support UNICODE?
Date: Tue, 8 Jun 2004 13:42:37 -0400
Message-ID: <38BCC8D26B88894DB8D921BD95121CAC09937D@MERC23.na.sas.com>
Thread-Topic: Does Lucene support UNICODE?
Thread-Index: AcRNJEtwOXQgL1LOSM6NiDU+TTOMEgAWxaIw
From: "Eric Isakson" <Eric.Isakson@sas.com>
To: "Lucene Users List" <lucene-user@jakarta.apache.org>

org.apache.lucene.demo.FileDocument.Document(File) is invoked from =
IndexFiles and does:

    Reader reader =3D new BufferedReader(new InputStreamReader(is));

Notice that the InputStreamReader does not specify an encoding so your =
default encoding is being used.

You should probably write your own glue to create your index that knows =
how to properly read your files in the appropriate encoding.

Also, please don't cross post to the dev and user lists, this is an =
appropriate question for the user list.

Eric

-----Original Message-----
From: Satish Kagathare [mailto:satishk@it.iitb.ac.in]=20
Sent: Tuesday, June 08, 2004 3:07 AM
To: Lucene Users List; lucene-dev@jakarta.apache.org
Subject: Does Lucene support UNICODE?


Hello,

Does Lucene support UNICODE search and indexing of UNICODE=20
data(especially..Devnagari unicode data)?
Does it make any difference between utf-8 & utf-16 unicode docs? Bcoz=20
java strings supports utf-16.

Bcoz i tried indexing(using indexFiles & indexHTML from lucene Demo)=20
devnagari uni data(utf-8 & utf-16) & seraching for query using tomcat,=20
but it shows only utf-8 files and also shows files which does not=20
contain query. Also It does not show summary of fetched docs in correct=20
format.

Also i have changed unicode range in HTMLparser.jj, StandardTokenizer.jj =
&=20
QueryParser.jj and analyzer while indexing and parsing query but it does =

not reflect any changes in output.
=20
shall i have to write my own analyzer for devnagari unicode data or=20
Standaranalyzer will work for any languages?

Or does it require more changes? Plz mention problems and solutions.=20

Thanks in advance
Satish Kagathara,
IIT Bombay.     =20

 =20


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org