Paul Elschot wrote:
>>Did you see my IDF question at the bottom of the original note? I'm
>>really curious why the square of IDF is used for Term and Phrase
>>queries, rather than just IDF. It seems like it might be a bug?
>
> I missed that.
> It has been discussed recently, but I don't remember the outcome,
> perhaps some else?
This has indeed been discussed before.
Lucene computes a dotproduct of a query vector and each document
vector. Weights in both vectors are normalized tf*idf, i.e.,
(tf*idf)/length. The dot product of vectors d and q is:
score(d,q) = sum over t of ( weight(t,q) * weight(t,d) )
Given this formulation, and the use of tf*idf weights, each component of
the sum has an idf^2 factor. That's just the way it works with dot
products of tf*idf/length vectors. It's not a bug. If folks don't like
it they can simply override Similarity.idf() to return sqrt(super()).
If someone can demonstrate that an alternate formulation produces
superior results for most applications, then we should of course change
the default implementation. But just noting that there's a factor which
is equal to idf^2 in each element of the sum does not do this.
Doug

To unsubscribe, email: lucenedevunsubscribe@jakarta.apache.org
For additional commands, email: lucenedevhelp@jakarta.apache.org
