Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Provalov, Ivan" <Ivan.Provalov@cengage.com>
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Date: Mon, 9 May 2011 17:32:56 -0400
Subject: Non-English Languages Search
Thread-Topic: Non-English Languages Search
Thread-Index: AQHMDpCoS92SIMrbLU2R6Fwy7pe3tQ==
Message-ID: <70EA5691BD59734784FF872CD1B9747A284F5D0B54@OHCINMBX01.corp.local>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

We are planning to ingest some non-English content into our application.  A=
ll content is OCR'ed and there are a lot of misspellings and garbage terms =
because of this.  Each document has one primary language with a some except=
ions (e.g. a few English terms mixed in with primarily non-English document=
 terms).

1. Does it make sense to mix two or more different Latin-based languages in=
 the same index directory in Lucene (e.g. Spanish/French/English)? =20
2. What about mixing Latin and non-Latin languages?  We ran tests on Englis=
h and Chinese collections mixed together and didn't see any negative impact=
 (precision/recall).  Any other potential issues?
3. Any recommendations for an Urdu analyzer?

Thank you,

Ivan Provalov=

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org