Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of mike.klaas@gmail.com
 designates 209.85.200.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer;
        b=YOu8bWJeheEQYmvAz7INbA8IDkZW8IOOxZgMPhHICRDfITxfE4/LKMyi3/9exP4otS8PhHiT/rUfFLUS9FcX7AlxWeVcyDqrPtKu4b1298R/9O/plnFxqXfg9sc8vDk0KZYtst+KPg9GYoQD7/dy9/yC3I9AA9zUWsy+bcgC1Gw=
Message-Id: <3E2220BE-A236-4290-9307-97FAF575BEA0@gmail.com>
From: Mike Klaas <mike.klaas@gmail.com>
To: solr-user@lucene.apache.org
In-Reply-To: <1467c2400805051328u589b1db3y7521e846dd7c883f@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v919.2)
Subject: Re: multi-language searching with Solr
Date: Tue, 6 May 2008 15:43:41 -0700
References: <1467c2400805050727p9914f62o9b8dacb840f1ee03@mail.gmail.com>
 <908893006339C0409519E4065DF3B24903188643@mailserver.ualibrary.ualberta.ca>
 <1467c2400805051328u589b1db3y7521e846dd7c883f@mail.gmail.com>

On 5-May-08, at 1:28 PM, Eli K wrote:

> Wouldn't this impact both indexing and search performance and the size
> of the index?
> It is also probable that I will have more then one free text fields
> later on and with at least 20 languages this approach does not seem
> very manageable.  Are there other options for making this work with
> stemming?

If you want stemming, then you have to execute one query per language =20=

anyway, since the stemming will be different in every language.

This is a fundamental requirement: you somehow need to track the =20
language of every token if you want correct multi-language stemming.  =20=

The easiest way to do this would be to split each language into its =20
own field.  But there are other options: you could prefix every =20
indexed token with the language:

en:The en:quick en:brown en:fox en:jumped ...
fr:Le fr:brun fr:renard fr:vite fr:a fr:saut=E9 ...

Separate fields seems easier to me, though.

-Mike=