lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J.Zhu" <>
Subject RE: Include BM25 in Lucene?
Date Tue, 17 Oct 2006 16:03:01 GMT
Hi, Vic,

Unfortunately BM25 uses IDF as well so splitting documents across
machines will also affect it. How about storing these as global
statistical data for sharing the search on these machines?

The equation of BM25 is clearly stated in Robertson's paper "Simple,
proven approaches to text retrieval"
( as follows.

CW (i,j) = [ CFW (i) * TF (i,j) * (K1+1) ] /[ K1 * ( (1-b) + (b * (NDL
(j)) ) ) + TF (i,j) ]
CFW(i) is collection frequency weight of term i, TF(i,j) is term
frequency of term i, NDL(j) is the normalized document length of
document j, and K1 and b are tuning constants. The details are in the

Univ. of Amsterdam has provided a downloadable version of a language
modelling version of Lucene. Their language model is not BM25 but is
quite similar in nature. The version is at:

I have worked on their version a bit, they have created new classes:
TermQueryLanguageModel, TermScorerLanguageModel,
IndexSearcherLanguageModel, LanguageModelIndexReader etc. I think their
work can be a basis.


-----Original Message-----
From: Vic Bancroft [] 
Sent: 17 October 2006 13:44
Subject: Re: Include BM25 in Lucene?

J.Zhu wrote:

>If I would like to contribute, what should I do? I am not a good Java 
>developer myself though. Can I work with someone also interested?
In some of my group's usage of lucene over large document collections,
we have split the documents across several machines.  This has lead to a
concern of whether the inverse document frequency was appropriate, since
the score seems to be dependant on the partioning of documents over
indexing hosts.  We have not formulated an experiment to determine if it
seriously effects our results, though it has been discussed.

If someone could elaborate how BM25 or some DFR algorithm would differ
from what (TF/IDF) is implemented in lucene, I would be willing to help
translate that into java as an indexing/searching option . . .


"The future is here. It's just not evenly distributed yet."
 -- William Gibson, quoted by Whitfield Diffie

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message