Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Subject: Skewed IDF in multi lingual index
From: =?utf-8?Q?Markus_Jelsma?= <markus.jelsma@openindex.io>
To: =?utf-8?Q?solr-user=40lucene=2Eapache=2Eorg?=
  <solr-user@lucene.apache.org>
Date: Thu, 8 Nov 2012 16:13:41 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-Id: <zarafa.509bda35.28f2.41b67d9f56419902@mail.openindex.io>

Hi,

We're testing a large multi lingual index with _LANG fields for each language and using dismax to query them all. Users provide, explicit or implicit, language preferences that we use for either additive or multiplicative boosting on the language of the document. However, additive boosting is not adequate because it cannot overcome the extremely high IDF values for the same word in another language so regardless of the the preference, foreign documents are returned. Multiplicative boosting solves this problem but has the other downside as it doesn't allow us with standard qf=3Dfield^boost to prefer documents in another language above the preferred language because the multiplicative is so strong. We do use the def function (boost=3Ddef(query($qq),.3)) to prevent one boost query to return 0 and thus a product of 0 for all boost queries. But it doesn't help that much

This all comes down to IDF differences between the languages, even common words such as country names like `india` show large differences in IDF. Is here anyone with some hints or experiences to share about skewed IDF in such an index=3F

Thanks,
Markus