Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
Message-ID: <453531ED.9080302@manawiz.com>
Date: Tue, 17 Oct 2006 09:41:33 -1000
From: Chuck Williams <chuck@manawiz.com>
Organization: Manawiz
User-Agent: Thunderbird 1.5.0.7 (X11/20060927)
MIME-Version: 1.0
To: java-dev@lucene.apache.org
Subject: Re: Include BM25 in Lucene?
References: <E0FEA5DF00E59E409F90C854A1B45BAA014704FB@EPPING-EVS1.open.ac.uk>
 <4534D017.7050100@america.net>
In-Reply-To: <4534D017.7050100@america.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Vic Bancroft wrote on 10/17/2006 02:44 AM:
> In some of my group's usage of lucene over large document collections,
> we have split the documents across several machines.  This has lead to
> a concern of whether the inverse document frequency was appropriate,
> since the score seems to be dependant on the partioning of documents
> over indexing hosts.  We have not formulated an experiment to
> determine if it seriously effects our results, though it has been
> discussed.

What version of Lucene are you using?  Are you using
ParallelMultiSearcher to manage the distributed indexes or have you
implemented your own mechanism?  There was a bug a couple years ago, in
the 1.4.3 version as I recall, where ParallelMultiSearcher was not
computing df's appropriately, but that has been fixed for a long time
now.  The df's are the sum of the df's from each distributed index and
thus are independent of the partitioning.

Chuck


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org